Issues upgrading on Kubernetes 1.25 - bump the Postgres (and possibly other) chart dependencies? #387

New Issue

2022-12-28T13:47:27Z

sseneca commented

2022-12-28 13:47:27 +00:00

(Migrated from gitea.com)

My Gitea won't update, with the following error:

Helm upgrade failed: resource mapping not found for name: "gitea-postgresql" namespace: "gitea" from "": no matches for kind "PodSecurityPolicy" in version "policy/v1beta1"

The version of the Postgres chart used is pretty old, I'm thinking it's probably time to bump it? That may also explain why it's trying to look for PodSecurityPolicy, even though they've been removed from Kubernetes (deprecated in 1.21, removed in 1.25, which is where I've been seeing these issues).

My Gitea won't update, with the following error: ``` Helm upgrade failed: resource mapping not found for name: "gitea-postgresql" namespace: "gitea" from "": no matches for kind "PodSecurityPolicy" in version "policy/v1beta1" ``` The [version of the Postgres chart](https://gitea.com/gitea/helm-chart/src/branch/main/Chart.yaml#L44) used is pretty old, I'm thinking it's probably time to bump it? That may also explain why it's trying to look for `PodSecurityPolicy`, even though they've been removed from Kubernetes (deprecated in 1.21, removed in 1.25, which is where I've been seeing these issues).

pat-s commented

2022-12-28 16:03:26 +00:00

(Migrated from gitea.com)

Thanks for the info!

One can surely think about bumping these versions (esp. for compat with k8s 1.25) though I am not sure we can provide all the required testing for various k8s versions constantly.
Most large cloud providers are at 1.24 now with 1.25 coming in Q1 2023.

In your case, you seem to have a conflict between k8s and and old postgres (chart) version. I guess it's your turn here bumping the version for compatibility for the moment. I guess the easiest might be to stay on the respective major version and just bump the to the latest patch version.

My personal opinion on this matter is: I would avoid depending on dependencies of charts if these are crucial elements like databases. A managed DB service solves a lot of headaches - or at least a central DB that you use for multiple services.

@justusbunsi What is your opinion here? I could also see a solution that sets all of these to :latest and adds a warning into the README that we do not recommend relying on this config for production installs?

Another, more drastic one, would be to force users to connect to an external DB and not even ship a fallback dependency to avoid future pains (or only ship one for testing purposes). I sometimes see this practice in other charts, i.e. just requiring a successful connection to an external DB in the first place.

Thanks for the info! One can surely think about bumping these versions (esp. for compat with k8s 1.25) though I am not sure we can provide all the required testing for various k8s versions constantly. Most large cloud providers are at 1.24 now with 1.25 coming in Q1 2023. In your case, you seem to have a conflict between k8s and and old postgres (chart) version. I guess it's your turn here bumping the version for compatibility for the moment. I guess the easiest might be to stay on the respective major version and just bump the to the latest patch version. My personal opinion on this matter is: I would avoid depending on dependencies of charts if these are crucial elements like databases. A managed DB service solves a lot of headaches - or at least a central DB that you use for multiple services. @justusbunsi What is your opinion here? I could also see a solution that sets all of these to `:latest` and adds a warning into the README that we do not recommend relying on this config for production installs? Another, more drastic one, would be to force users to connect to an external DB and not even ship a fallback dependency to avoid future pains (or only ship one for testing purposes). I sometimes see this practice in other charts, i.e. just requiring a successful connection to an external DB in the first place.

sseneca commented

2022-12-28 17:19:54 +00:00

(Migrated from gitea.com)

Looking at the Nextcloud Helm chart (which similarly bundles Postgres, MariaDB, etc), this is what they do:

dependencies:
  - name: postgresql
    version: 11.6.*
    repository: https://charts.bitnami.com/bitnami
    condition: postgresql.enabled
  - name: mariadb
    version: 11.0.*
    repository: https://charts.bitnami.com/bitnami
    condition: mariadb.enabled

These deps are then occasionally bumped, which also prompts a bump of the major version of the chart.

Users of the chart should know that a new major version of the Gitea Helm chart means they should check the changelog for any breaking changes. In the changelog/release notes of said newer major version, you could note that the Postgres dependency was bumped. IMO this is quite reasonable.

Looking at the Nextcloud Helm chart (which similarly bundles Postgres, MariaDB, etc), [this is what they do](https://github.com/nextcloud/helm/blob/master/charts/nextcloud/Chart.yaml#L26): ```yaml dependencies: - name: postgresql version: 11.6.* repository: https://charts.bitnami.com/bitnami condition: postgresql.enabled - name: mariadb version: 11.0.* repository: https://charts.bitnami.com/bitnami condition: mariadb.enabled ``` These deps are then [occasionally bumped](https://github.com/nextcloud/helm/commit/3fd13733b38e3df9f2923e42f9eb7e6ee37a96e4), which also prompts a bump of the major version of the chart. Users of the chart should know that a new major version of the Gitea Helm chart means they should check the changelog for any breaking changes. In the changelog/release notes of said newer major version, you could note that the Postgres dependency was bumped. IMO this is quite reasonable.

pat-s commented

2022-12-30 16:28:20 +00:00

(Migrated from gitea.com)

Users of the chart should know that a new major version of the Gitea Helm chart means they should check the changelog for any breaking changes. In the changelog/release notes of said newer major version, you could note that the Postgres dependency was bumped. IMO this is quite reasonable.

What you describe here is how an potential upgrade procedure would work in terms of NEWS etc. This is pretty clear and actually not the "problem" and I think we all agree on the procedure.

The questions are more generic:

In which interval should these bumps happen? Do we always just wait until some error is reported like right now?
How do we test them upfront? I.e. while it is likely everything will "just work" I/we don't have the time and capabilities to easily test everything against a specific old postgres version and verify that "things are good". And doing so in CI is also not that easy.
We cannot just go with the lower limit Gitea recommends itself because of the k8s issues you described. Hence I wonder if :latest would be a better pick here, being aware of all the issues that come when working with a :latest tag
We could just follow other projects like Nextcloud in the hope they have a "concept" for testing these versions but actually that does sound a bit like a "cheap" "copying" approach

All ideas are welcome!
I think this is a good issues to think a bit more about the bigger picture rather than doing a "quick fix" - this can already easily be done by changing the image tag on the user level.

> Users of the chart should know that a new major version of the Gitea Helm chart means they should check the changelog for any breaking changes. In the changelog/release notes of said newer major version, you could note that the Postgres dependency was bumped. IMO this is quite reasonable. What you describe here is how an potential upgrade procedure would work in terms of NEWS etc. This is pretty clear and actually not the "problem" and I think we all agree on the procedure. The questions are more generic: - In which interval should these bumps happen? Do we always just wait until some error is reported like right now? - How do we test them upfront? I.e. while it is likely everything will "just work" I/we don't have the time and capabilities to easily test everything against a specific old postgres version and verify that "things are good". And doing so in CI is also not that easy. - We cannot just go with the lower limit Gitea recommends itself because of the k8s issues you described. Hence I wonder if `:latest` would be a better pick here, being aware of all the issues that come when working with a `:latest` tag - We could just follow other projects like Nextcloud in the hope they have a "concept" for testing these versions but actually that does sound a bit like a "cheap" "copying" approach All ideas are welcome! I think this is a good issues to think a bit more about the bigger picture rather than doing a "quick fix" - this can already easily be done by changing the image tag on the user level.

mattrpav commented

2023-01-03 16:22:19 +00:00

(Migrated from gitea.com)

Helm supports marking a template with kubeVersion. That would be a way to set a semVer range for a given chart.

We are still looking for a best practice approach for managing the k8s api changes, such as this. Looking at separate charts (ugh) or using a macro to detect the kubernetes version and then adapt the chart accordingly.

IMO :latest is more problematic by default. Users can always override the image field if they want to in their own deployments.

Agreed, testing all combos of versions is not really practical, but best effort and then adapt based on feedback.

Helm is usually flexible enough that a lot of things can be hacked or templated over to get around issues, so users aren't generally 'stuck' and waiting for a release.

Helm supports marking a template with *kubeVersion*. That would be a way to set a semVer range for a given chart. We are still looking for a best practice approach for managing the k8s api changes, such as this. Looking at separate charts (ugh) or using a macro to detect the kubernetes version and then adapt the chart accordingly. IMO :latest is more problematic by default. Users can always override the image field if they want to in their own deployments. Agreed, testing all combos of versions is not really practical, but best effort and then adapt based on feedback. Helm is usually flexible enough that a lot of things can be hacked or templated over to get around issues, so users aren't generally 'stuck' and waiting for a release.

sseneca commented

2023-01-04 11:49:06 +00:00

(Migrated from gitea.com)

Not sure if I'm misunderstanding, but to be clear, I think my issue is not with the image version of Postgres which I know I can bump, but with the chart version of the Postgres chart bundled with the Gitea chart which (as far as I know?) I cannot bump.

I think, because the Bitnami Postres chart is set to an old version, it doesn't properly implement not trying to install a PodSecurityPolicy, which my cluster does not have (it was removed in k8s 1.25). So I think I need a newer version of the Postgres Helm chart that does not try to do anything with the PSP, because atm it is refusing to Helm upgrade my Gitea install (the aforementioned no matches for kind "PodSecurityPolicy" in version "policy/v1beta1). Hopefully that made a bit more sense.

Not sure if I'm misunderstanding, but to be clear, I think my issue is *not* with the image version of Postgres which I know I can bump, but with the *chart version* of the Postgres chart bundled with the Gitea chart which (as far as I know?) I cannot bump. I *think*, because the Bitnami Postres chart is set to an old version, it doesn't properly implement not trying to install a PodSecurityPolicy, which my cluster does not have (it was removed in k8s 1.25). So I think I need a newer version of the Postgres Helm chart that does not try to do anything with the PSP, because atm it is refusing to Helm upgrade my Gitea install (the aforementioned `no matches for kind "PodSecurityPolicy" in version "policy/v1beta1`). Hopefully that made a bit more sense.

mattrpav commented

2023-01-04 14:19:14 +00:00

(Migrated from gitea.com)

The postgresql chart has a config override (confirmed in gitea helm chart v5.0.9).

apiVersion: {{ include "podsecuritypolicy.apiVersion" . }}

try adding that property to the 'postgres:' section of your values.yaml to override.

The wider conversation is about how to manage k8s api change (& deprecation) and dependency versions.

I found that Helm has some built-ins for kubernetes version management. This looks to be a solid approach.

{{- if semverCompare ">=1.19-0" .Capabilities.KubeVersion.GitVersion -}}
apiVersion: networking.k8s.io/v1
{{- else if semverCompare ">=1.14-0" .Capabilities.KubeVersion.GitVersion -}}
apiVersion: networking.k8s.io/v1beta1
{{- else -}}
apiVersion: extensions/v1beta1
{{- end }}

h/t: https://renehernandez.io/snippets/customize-api-version-helm-templates/

The postgresql chart has a config override (confirmed in gitea helm chart v5.0.9). ``` apiVersion: {{ include "podsecuritypolicy.apiVersion" . }} ``` try adding that property to the 'postgres:' section of your values.yaml to override. The wider conversation is about how to manage k8s api change (& deprecation) and dependency versions. I found that Helm has some built-ins for kubernetes version management. This looks to be a solid approach. ``` {{- if semverCompare ">=1.19-0" .Capabilities.KubeVersion.GitVersion -}} apiVersion: networking.k8s.io/v1 {{- else if semverCompare ">=1.14-0" .Capabilities.KubeVersion.GitVersion -}} apiVersion: networking.k8s.io/v1beta1 {{- else -}} apiVersion: extensions/v1beta1 {{- end }} ``` h/t: https://renehernandez.io/snippets/customize-api-version-helm-templates/

pat-s commented

2023-01-05 09:09:44 +00:00

(Migrated from gitea.com)

Not sure if I'm misunderstanding, but to be clear, I think my issue is not with the image version of Postgres which I know I can bump, but with the chart version of the Postgres chart bundled with the Gitea chart which (as far as I know?) I cannot bump

Ah yes, you cannot (easily). I think we need to upgrade this one to have proper k8s support.

Looking at https://docs.bitnami.com/kubernetes/infrastructure/postgresql/administration/upgrade/ there might be a bit of work needed but we can just see how it goes.

@sseneca I can create a branch with 11.x and you could potentially test this with helm-git if that helps? We can also try going to 12.x afterwards if you're interested.

@mattrpav Thanks for the info! Though I think putting the time in to go with the latest release of the chart might make more sense in contrast to adding (complex) semver conditions for k8s versions?

> Not sure if I'm misunderstanding, but to be clear, I think my issue is not with the image version of Postgres which I know I can bump, but with the chart version of the Postgres chart bundled with the Gitea chart which (as far as I know?) I cannot bump Ah yes, you cannot (easily). I think we need to upgrade this one to have proper k8s support. Looking at https://docs.bitnami.com/kubernetes/infrastructure/postgresql/administration/upgrade/ there might be a bit of work needed but we can just see how it goes. @sseneca I can create a branch with 11.x and you could potentially test this with [helm-git](https://github.com/aslafy-z/helm-git) if that helps? We can also try going to 12.x afterwards if you're interested. @mattrpav Thanks for the info! Though I think putting the time in to go with the latest release of the chart might make more sense in contrast to adding (complex) semver conditions for k8s versions?

sseneca commented

2023-01-10 14:57:24 +00:00

(Migrated from gitea.com)

Hey, really bad timing I know but just before that branch went up, I decided to just downgrade my cluster, delete the PSP, upgrade Gitea, and then upgrade my cluster again. This worked fine, but I unfortunately can't test your solution.

pat-s commented

2023-02-26 12:55:50 +00:00

(Migrated from gitea.com)

See #391

pat-s commented

2023-04-07 11:01:55 +00:00

(Migrated from gitea.com)

Assuming this has been solved by the PG dependency bump.

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: lunny/helm-chart#387