lunny/helm-chart
1
0
Fork 0
Code Issues 36 Pull Requests 14 Actions Packages Projects Releases 93 Wiki Activity

UPGRADE FAILED: cannot patch "gitea-postgresql" with kind StatefulSet #452

New Issue
Open
opened 2023-05-31 05:17:41 +00:00 by pi3ch · 6 comments
pi3ch commented 2023-05-31 05:17:41 +00:00 (Migrated from gitea.com)
Copy Link

First time having issue with gitea upgrade.

I was on v1.18.1 gitea and trying to upgrade to v1.19.3 using same values as before.

> helm upgrade gitea gitea-charts/gitea --namespace gitea --values values.yaml
coalesce.go:175: warning: skipped value for memcached.initContainers: Not a table.
Error: UPGRADE FAILED: execution error at (gitea/charts/postgresql/templates/secrets.yaml:5:24): 
PASSWORDS ERROR: The secret "gitea-postgresql" does not contain the key "postgres-password"

> helm upgrade gitea gitea-charts/gitea --namespace gitea --values values.yaml
coalesce.go:175: warning: skipped value for memcached.initContainers: Not a table.
Error: UPGRADE FAILED: execution error at (gitea/charts/postgresql/templates/secrets.yaml:18:16): 
PASSWORDS ERROR: The secret "gitea-postgresql" does not contain the key "password"

I went and patch gitea-postgresql secret and copy the old key to the new name as required. Reran the upgrade and got:

Error: UPGRADE FAILED: cannot patch "gitea-postgresql" with kind StatefulSet: StatefulSet.apps "gitea-postgresql" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', 'updateStrategy', 'persistentVolumeClaimRetentionPolicy' and 'minReadySeconds' are forbidden

I can see gitea 1.19 is working and db tables are upgraded but the above error tell me postgres image is not updated.

Any pointers?

First time having issue with gitea upgrade. I was on v1.18.1 gitea and trying to upgrade to v1.19.3 using same values as before. ``` > helm upgrade gitea gitea-charts/gitea --namespace gitea --values values.yaml coalesce.go:175: warning: skipped value for memcached.initContainers: Not a table. Error: UPGRADE FAILED: execution error at (gitea/charts/postgresql/templates/secrets.yaml:5:24): PASSWORDS ERROR: The secret "gitea-postgresql" does not contain the key "postgres-password" > helm upgrade gitea gitea-charts/gitea --namespace gitea --values values.yaml coalesce.go:175: warning: skipped value for memcached.initContainers: Not a table. Error: UPGRADE FAILED: execution error at (gitea/charts/postgresql/templates/secrets.yaml:18:16): PASSWORDS ERROR: The secret "gitea-postgresql" does not contain the key "password" ``` I went and patch `gitea-postgresql` secret and copy the old key to the new name as required. Reran the upgrade and got: ``` Error: UPGRADE FAILED: cannot patch "gitea-postgresql" with kind StatefulSet: StatefulSet.apps "gitea-postgresql" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', 'updateStrategy', 'persistentVolumeClaimRetentionPolicy' and 'minReadySeconds' are forbidden ``` I can see gitea 1.19 is working and db tables are upgraded but the above error tell me postgres image is not updated. Any pointers?
justusbunsi commented 2023-05-31 05:31:44 +00:00 (Migrated from gitea.com)
Copy Link

From which Helm chart version did you update? And to which Chart version? You might have to redeploy postgres due to dependency updates.

From which Helm chart version did you update? And to which Chart version? You might have to redeploy postgres due to dependency updates.
pat-s commented 2023-05-31 06:35:28 +00:00 (Migrated from gitea.com)
Copy Link

Jup make sure to check https://gitea.com/gitea/helm-chart#to-8-0-0 - requires a manual DB upgrade of the postgres pod

We had another user before in discord who reported that. steps are:

  1. backup your PG 11
  2. Start with PG 15
  3. Restore backup

Or - recommended alternative - go with an external DB that is not managed by the chart for which you have more control :)

Jup make sure to check https://gitea.com/gitea/helm-chart#to-8-0-0 - requires a manual DB upgrade of the postgres pod We had another user before in discord who reported that. steps are: 1. backup your PG 11 2. Start with PG 15 3. Restore backup Or - recommended alternative - go with an external DB that is not managed by the chart for which you have more control :)
pi3ch commented 2023-05-31 13:14:35 +00:00 (Migrated from gitea.com)
Copy Link

OK. so I endded up in a bit of drama. Gitea was upgraded to 1.19 and hence db tables has been upgraded on PG 11.

Uninstalled and reinstalled helm chart.

PG 15 couldn't load due to old version of data directory. PG 15 requires manual update from PG 11. This requires configuring and running manual jobs.

Downgraded to gitea 1.18. DB came up but gitea didn't as there was changes to DB table.

Upgraded to Gitea 1.19 to get gitea running. Now it runs with PG 11 but helm chart fails (errors in my original post).

Using external DBs is not an option for me.

Couldn't upgrade 11 to 15 be scripted as part of the chart? I think a lot of people gonna have similar situation like me as PG upgrade is not straight forward given everything was managed by helm chart before.

Regarding the release note, I think a "breaking change" notice should be included and it should clearly state "DB requires manual upgrade".

OK. so I endded up in a bit of drama. Gitea was upgraded to 1.19 and hence db tables has been upgraded on PG 11. Uninstalled and reinstalled helm chart. PG 15 couldn't load due to old version of `data` directory. PG 15 requires manual update from PG 11. This requires configuring and running manual jobs. Downgraded to gitea 1.18. DB came up but gitea didn't as there was changes to DB table. Upgraded to Gitea 1.19 to get gitea running. Now it runs with PG 11 but helm chart fails (errors in my original post). Using external DBs is not an option for me. Couldn't upgrade 11 to 15 be scripted as part of the chart? I think a lot of people gonna have similar situation like me as PG upgrade is not straight forward given everything was managed by helm chart before. Regarding the release note, I think a "breaking change" notice should be included and it should clearly state "DB requires manual upgrade".
pat-s commented 2023-05-31 15:04:30 +00:00 (Migrated from gitea.com)
Copy Link

Sorry to hear you've faced issues.

Regarding the release note, I think a "breaking change" notice should be included and it should clearly state "DB requires manual upgrade".

A major version bump implies breaking changes and the update of the PG version is mentioned there. There is no "simple upgrade path" from a dockerized DB version to another major version that could just be done by the helm chart (https://www.cloudytuts.com/tutorials/docker/how-to-upgrade-postgresql-in-docker-and-kubernetes/). We also can't stick with an old PG version forever.

PRs to improve the update experience are always welcome - we all do this with "best effort" in our spare time.

Using external DBs is not an option for me.

You can always run a standalone PG besides the gitea container that is not managed by the helm-chart. It doesn't need to be a managed-DB in the cloud.

Sorry to hear you've faced issues. > Regarding the release note, I think a "breaking change" notice should be included and it should clearly state "DB requires manual upgrade". A major version bump implies breaking changes and the update of the PG version is mentioned there. There is no "simple upgrade path" from a dockerized DB version to another major version that could just be done by the helm chart (https://www.cloudytuts.com/tutorials/docker/how-to-upgrade-postgresql-in-docker-and-kubernetes/). We also can't stick with an old PG version forever. PRs to improve the update experience are always welcome - we all do this with "best effort" in our spare time. > Using external DBs is not an option for me. You can always run a standalone PG besides the gitea container that is not managed by the helm-chart. It doesn't need to be a managed-DB in the cloud.
pi3ch commented 2023-06-01 05:09:32 +00:00 (Migrated from gitea.com)
Copy Link

OK, I managed to get things working with PG 15 (it was also okay to continue using PG 11 but helm upgrade fail).

It was not a straight forward upgrade due to some issue which I explain below.

Create a SQL dump of your old gitea database: kubectl exec -it gitea-postgresql-0 -- pg_dumpall -U postgres > /tmp/gitea.sql

NOTE: If you don't know the postgres password, run sed -ibak 's/^\([^#]*\)md5/\1trust/g' /opt/bitnami/postgresql/conf/pg_hba.conf && pg_ctl reload to remove the password requirement locally (see https://docs.bitnami.com/virtual-machine/infrastructure/postgresql/administration/change-reset-password/)

Copy it locally kubectl cp ....

Edit the gitea.sql and comment lines where it CREATE ROLE postgres and ALTER ROLE postgres. Change ALTER DATABASE gitea OWNER TO postgres to ALTER DATABASE gitea OWNER TO gitea If you don't you may lose access to postgres (I couldn't find the postgres password that was used in PG 11).

Create a new postgres db. Many options here.

Copy gitea.sql to your new postgres: kubectl cp ...

Restore the backup to your new postgres: kubectl exec -it gitea-postgres-new -- psql -U postgres < /tmp/gitea.sql

Modify gitea helm chart to disable postgres but add your new postgres credentials:

postgresql:
  enabled: false
  global:
    postgresql:
      auth:
        password: <gitea db password - keep it as it was before usually all gitea> 
        database: <gitea db name>
        username: <gitea db username>

BUG: if you postgres service name is different, gitea initilisation script still points to old postgres service name. You need to update K8S secret > gitea-inline-config > database with name of your new postgres service. If you don't do this, gitea pod will never come up when you upgrade (attn @pat-s)

Apply helm upgrade. Gitea should come up with no issue.

BUG: I found webhooks fail in DB migrations:

STATEMENT:  CREATE DATABASE gitea WITH TEMPLATE = template0 ENCODING = 'UTF8' LC_COLLATE = 'en_US.UTF-8' LC_CTYPE = 'en_US.UTF-8';
2023-06-01 04:24:28.363 GMT [305] ERROR:  missing data for column "header_authorization_encrypted"
2023-06-01 04:24:28.363 GMT [305] CONTEXT:  COPY webhook, line 4: "627 90 0 f https://webhook-address/hook?secret=nXXXXXXXXXXXX POST 1 ..."
2023-06-01 04:24:28.363 GMT [305] STATEMENT:  COPY public.webhook (id, repo_id, org_id, is_system_webhook, url, http_method, content_type, secret, events, is_active, type, meta, last_status, created_unix, updated_unix, header_authorization_encrypted) FROM stdin;

Therefore, make sure to create the webhook again.

OK, I managed to get things working with PG 15 (it was also okay to continue using PG 11 but helm upgrade fail). It was not a straight forward upgrade due to some issue which I explain below. Create a SQL dump of your old gitea database: `kubectl exec -it gitea-postgresql-0 -- pg_dumpall -U postgres > /tmp/gitea.sql` NOTE: If you don't know the postgres password, run `sed -ibak 's/^\([^#]*\)md5/\1trust/g' /opt/bitnami/postgresql/conf/pg_hba.conf && pg_ctl reload` to remove the password requirement locally (see https://docs.bitnami.com/virtual-machine/infrastructure/postgresql/administration/change-reset-password/) Copy it locally `kubectl cp ....` Edit the `gitea.sql` and comment lines where it `CREATE ROLE postgres` and `ALTER ROLE postgres`. Change `ALTER DATABASE gitea OWNER TO postgres` to `ALTER DATABASE gitea OWNER TO gitea` If you don't you may lose access to postgres (I couldn't find the postgres password that was used in PG 11). Create a new postgres db. Many options here. Copy `gitea.sql` to your new postgres: `kubectl cp ...` Restore the backup to your new postgres: ` kubectl exec -it gitea-postgres-new -- psql -U postgres < /tmp/gitea.sql` Modify gitea helm chart to disable postgres but add your new postgres credentials: ``` postgresql: enabled: false global: postgresql: auth: password: <gitea db password - keep it as it was before usually all gitea> database: <gitea db name> username: <gitea db username> ``` BUG: if you postgres service name is different, gitea initilisation script still points to old postgres service name. You need to update K8S secret > gitea-inline-config > database with name of your new postgres service. If you don't do this, gitea pod will never come up when you upgrade (attn @pat-s) Apply helm upgrade. Gitea should come up with no issue. BUG: I found webhooks fail in DB migrations: ``` STATEMENT: CREATE DATABASE gitea WITH TEMPLATE = template0 ENCODING = 'UTF8' LC_COLLATE = 'en_US.UTF-8' LC_CTYPE = 'en_US.UTF-8'; 2023-06-01 04:24:28.363 GMT [305] ERROR: missing data for column "header_authorization_encrypted" 2023-06-01 04:24:28.363 GMT [305] CONTEXT: COPY webhook, line 4: "627 90 0 f https://webhook-address/hook?secret=nXXXXXXXXXXXX POST 1 ..." 2023-06-01 04:24:28.363 GMT [305] STATEMENT: COPY public.webhook (id, repo_id, org_id, is_system_webhook, url, http_method, content_type, secret, events, is_active, type, meta, last_status, created_unix, updated_unix, header_authorization_encrypted) FROM stdin; ``` Therefore, make sure to create the webhook again.
👍 1
pat-s commented 2023-06-02 10:22:15 +00:00 (Migrated from gitea.com)
Copy Link

@pi3ch Thanks for the detailed instructions! These could make it into an official document to help other users along the way. I could think of a simple .md document in a new directory docs/ in the repo titled postgres-upgrade-instructions.md or similar.

Would you be motivated to port your comment into a doc?

We're aware that the process is of conducting a major update of a DB is not a trivial one. However, everybody has a different setup and it's hard for us (and partly out-of-scope) to provide instructions how to maintain all dependencies in this chart.
That was also part of the motivation why we removed all the other DB types lately - just to have less issues WRT to dependencies.

And congrats on getting it working! 🎉 I am sure it took more than just a few minutes...

BUG: if you postgres service name is different, gitea initilisation script still points to old postgres service name. You need to update K8S secret > gitea-inline-config > database with name of your new postgres service. If you don't do this, gitea pod will never come up when you upgrade (attn @pat-s)

Is this due to how bitnami names it, i.e. caused by the PG chart update? Overall this sounds a bit as if a clean init of app.ini could help here, see #356 and others for more information (tldr, we're aware of it).

@pi3ch Thanks for the detailed instructions! These could make it into an official document to help other users along the way. I could think of a simple `.md` document in a new directory `docs/` in the repo titled `postgres-upgrade-instructions.md` or similar. Would you be motivated to port your comment into a doc? We're aware that the process is of conducting a major update of a DB is not a trivial one. However, everybody has a different setup and it's hard for us (and partly out-of-scope) to provide instructions how to maintain all dependencies in this chart. That was also part of the motivation why we removed all the other DB types lately - just to have less issues WRT to dependencies. And congrats on getting it working! 🎉 I am sure it took more than just a few minutes... > BUG: if you postgres service name is different, gitea initilisation script still points to old postgres service name. You need to update K8S secret > gitea-inline-config > database with name of your new postgres service. If you don't do this, gitea pod will never come up when you upgrade (attn @pat-s) Is this due to how bitnami names it, i.e. caused by the PG chart update? Overall this sounds a bit as if a clean init of `app.ini` could help here, see #356 and others for more information (tldr, we're aware of it).
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
has/backport
in progress
invalid
kind/breaking
kind/bug
kind/build
kind/dependency
kind/deployment
kind/docs
kind/enhancement
kind/feature
kind/lint
kind/proposal
kind/question
kind/refactor
kind/security
kind/testing
kind/translation
kind/ui
need/backport
priority/critical
priority/low
priority/maybe
priority/medium
reviewed/duplicate
reviewed/invalid
reviewed/wontfix
skip-changelog
status/blocked
status/needs-feedback
status/needs-reviews
status/wip
upstream/gitea
upstream/other
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: lunny/helm-chart#452
No description provided.
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?