Cannot access existing S3 bucket after upgrade from v8.3.0 to V9.0.0 #469

Closed
opened 2023-07-18 12:41:56 +00:00 by tobiasbp · 10 comments
tobiasbp commented 2023-07-18 12:41:56 +00:00 (Migrated from gitea.com)

Having upgraded from v8.3.0 to v9.0.0, I can no longer access the existing S3 bucket I used with v8.3.0. The bucket is in a Google cloud project. Any suggestions appreciated.

This is the storage config used:

    storage:
      MINIO_ACCESS_KEY_ID: ***
      MINIO_BUCKET: my-unique-bucket-name
      MINIO_ENDPOINT: storage.googleapis.com
      MINIO_INSECURE_SKIP_VERIFY: false
      MINIO_SECRET_ACCESS_KEY: ***
      MINIO_USE_SSL: true
      SERVE_DIRECT: false
      STORAGE_TYPE: minio

Here is the log:

kubectl -n git-backend logs pod/gitea-697958d46b-7gwt9

Defaulted container "gitea" out of: gitea, init-directories (init), init-app-ini (init), configure-gitea (init)
2023/07/18 12:16:51 cmd/web.go:223:runWeb() [I] Starting Gitea on PID: 7
2023/07/18 12:16:51 cmd/web.go:148:serveInstalled() [I] Gitea version: 1.20.0 built with GNU Make 4.4.1, go1.20.6 : bindata, timetzdata, sqlite, sqlite_unlock_notify
2023/07/18 12:16:51 cmd/web.go:149:serveInstalled() [I] App path: /usr/local/bin/gitea
2023/07/18 12:16:51 cmd/web.go:150:serveInstalled() [I] Work path: /data
2023/07/18 12:16:51 cmd/web.go:151:serveInstalled() [I] Custom path: /data/gitea
2023/07/18 12:16:51 cmd/web.go:152:serveInstalled() [I] Config file: /data/gitea/conf/app.ini
2023/07/18 12:16:51 cmd/web.go:153:serveInstalled() [I] Run mode: prod
2023/07/18 12:16:51 cmd/web.go:154:serveInstalled() [I] Prepare to run web server
2023/07/18 12:16:51 routers/init.go:112:InitWebInstalled() [I] Git version: 2.40.1, Wire Protocol Version 2 Enabled (home: /data/home)
2023/07/18 12:16:53 ...les/setting/cache.go:75:loadCacheFrom() [I] Cache Service Enabled
2023/07/18 12:16:53 ...les/setting/cache.go:90:loadCacheFrom() [I] Last Commit Cache Service Enabled
2023/07/18 12:16:53 ...s/setting/session.go:74:loadSessionFrom() [I] Session Service Enabled
2023/07/18 12:16:53 ...s/storage/storage.go:176:initAttachments() [I] Initialising Attachment storage with type: minio
2023/07/18 12:16:53 ...les/storage/minio.go:81:NewMinioStorage() [I] Creating Minio storage at storage.googleapis.com:gitea-stg-aqrk with base path attachments/
2023/07/18 12:16:53 routers/init.go:60:mustInit() [F] code.gitea.io/gitea/modules/storage.Init failed: The requested bucket name is not available. The bucket namespace is shared by all users of the system. Please select a different name and try again.
Having upgraded from v8.3.0 to v9.0.0, I can no longer access the existing S3 bucket I used with v8.3.0. The bucket is in a Google cloud project. Any suggestions appreciated. This is the storage config used: ``` storage: MINIO_ACCESS_KEY_ID: *** MINIO_BUCKET: my-unique-bucket-name MINIO_ENDPOINT: storage.googleapis.com MINIO_INSECURE_SKIP_VERIFY: false MINIO_SECRET_ACCESS_KEY: *** MINIO_USE_SSL: true SERVE_DIRECT: false STORAGE_TYPE: minio ``` Here is the log: ``` kubectl -n git-backend logs pod/gitea-697958d46b-7gwt9 Defaulted container "gitea" out of: gitea, init-directories (init), init-app-ini (init), configure-gitea (init) 2023/07/18 12:16:51 cmd/web.go:223:runWeb() [I] Starting Gitea on PID: 7 2023/07/18 12:16:51 cmd/web.go:148:serveInstalled() [I] Gitea version: 1.20.0 built with GNU Make 4.4.1, go1.20.6 : bindata, timetzdata, sqlite, sqlite_unlock_notify 2023/07/18 12:16:51 cmd/web.go:149:serveInstalled() [I] App path: /usr/local/bin/gitea 2023/07/18 12:16:51 cmd/web.go:150:serveInstalled() [I] Work path: /data 2023/07/18 12:16:51 cmd/web.go:151:serveInstalled() [I] Custom path: /data/gitea 2023/07/18 12:16:51 cmd/web.go:152:serveInstalled() [I] Config file: /data/gitea/conf/app.ini 2023/07/18 12:16:51 cmd/web.go:153:serveInstalled() [I] Run mode: prod 2023/07/18 12:16:51 cmd/web.go:154:serveInstalled() [I] Prepare to run web server 2023/07/18 12:16:51 routers/init.go:112:InitWebInstalled() [I] Git version: 2.40.1, Wire Protocol Version 2 Enabled (home: /data/home) 2023/07/18 12:16:53 ...les/setting/cache.go:75:loadCacheFrom() [I] Cache Service Enabled 2023/07/18 12:16:53 ...les/setting/cache.go:90:loadCacheFrom() [I] Last Commit Cache Service Enabled 2023/07/18 12:16:53 ...s/setting/session.go:74:loadSessionFrom() [I] Session Service Enabled 2023/07/18 12:16:53 ...s/storage/storage.go:176:initAttachments() [I] Initialising Attachment storage with type: minio 2023/07/18 12:16:53 ...les/storage/minio.go:81:NewMinioStorage() [I] Creating Minio storage at storage.googleapis.com:gitea-stg-aqrk with base path attachments/ 2023/07/18 12:16:53 routers/init.go:60:mustInit() [F] code.gitea.io/gitea/modules/storage.Init failed: The requested bucket name is not available. The bucket namespace is shared by all users of the system. Please select a different name and try again. ```
viceice commented 2023-07-18 13:23:47 +00:00 (Migrated from gitea.com)

You need to remove all storage types from the sections and only leace the storage general section

image

You need to remove all storage types from the sections and only leace the storage general section ![image](/attachments/2c80fddc-1f2d-4093-94ce-eae04f08c94a)
tobiasbp commented 2023-07-18 13:32:05 +00:00 (Migrated from gitea.com)

UPDATE: I have realized that specifying MINIO_LOCATION is only needed when a bucket is to be created. I will remove it from my config, as my bucket exists.


Original post:

My bucket is not in the default location us-east-1. I have added this to fix it (How come it worked without in v8.3.0?):

MINIO_LOCATION: europe-north1

Gitea now reports a permission problem. The credentials are the ones I successfully used in v8.3.0:

Defaulted container "gitea" out of: gitea, init-directories (init), init-app-ini (init), configure-gitea (init). 

2023/07/18 13:20:38 cmd/web.go:223:runWeb() [I] Starting Gitea on PID: 7
2023/07/18 13:20:38 cmd/web.go:148:serveInstalled() [I] Gitea version: 1.20.0 built with GNU Make 4.4.1, go1.20.6 : bindata, timetzdata, sqlite, sqlite_unlock_notify
2023/07/18 13:20:38 cmd/web.go:149:serveInstalled() [I] App path: /usr/local/bin/gitea
2023/07/18 13:20:38 cmd/web.go:150:serveInstalled() [I] Work path: /data
2023/07/18 13:20:38 cmd/web.go:151:serveInstalled() [I] Custom path: /data/gitea
2023/07/18 13:20:38 cmd/web.go:152:serveInstalled() [I] Config file: /data/gitea/conf/app.ini
2023/07/18 13:20:38 cmd/web.go:153:serveInstalled() [I] Run mode: prod
2023/07/18 13:20:38 cmd/web.go:154:serveInstalled() [I] Prepare to run web server
2023/07/18 13:20:38 routers/init.go:112:InitWebInstalled() [I] Git version: 2.40.1, Wire Protocol Version 2 Enabled (home: /data/home)
2023/07/18 13:20:39 ...les/setting/cache.go:75:loadCacheFrom() [I] Cache Service Enabled
2023/07/18 13:20:39 ...les/setting/cache.go:90:loadCacheFrom() [I] Last Commit Cache Service Enabled
2023/07/18 13:20:39 ...s/setting/session.go:74:loadSessionFrom() [I] Session Service Enabled
2023/07/18 13:20:39 ...s/storage/storage.go:176:initAttachments() [I] Initialising Attachment storage with type: minio
2023/07/18 13:20:39 ...les/storage/minio.go:81:NewMinioStorage() [I] Creating Minio storage at storage.googleapis.com:gitea-stg-aqrk with base path attachments/
2023/07/18 13:20:40 routers/init.go:60:mustInit() [F] code.gitea.io/gitea/modules/storage.Init failed: permission denied
UPDATE: I have realized that specifying _MINIO_LOCATION_ is only needed when a bucket is to be created. I will remove it from my config, as my bucket exists. ---- Original post: My bucket is not in the default location _us-east-1_. I have added this to fix it (How come it worked without in v8.3.0?): ``` MINIO_LOCATION: europe-north1 ``` Gitea now reports a permission problem. The credentials are the ones I successfully used in v8.3.0: ``` Defaulted container "gitea" out of: gitea, init-directories (init), init-app-ini (init), configure-gitea (init). 2023/07/18 13:20:38 cmd/web.go:223:runWeb() [I] Starting Gitea on PID: 7 2023/07/18 13:20:38 cmd/web.go:148:serveInstalled() [I] Gitea version: 1.20.0 built with GNU Make 4.4.1, go1.20.6 : bindata, timetzdata, sqlite, sqlite_unlock_notify 2023/07/18 13:20:38 cmd/web.go:149:serveInstalled() [I] App path: /usr/local/bin/gitea 2023/07/18 13:20:38 cmd/web.go:150:serveInstalled() [I] Work path: /data 2023/07/18 13:20:38 cmd/web.go:151:serveInstalled() [I] Custom path: /data/gitea 2023/07/18 13:20:38 cmd/web.go:152:serveInstalled() [I] Config file: /data/gitea/conf/app.ini 2023/07/18 13:20:38 cmd/web.go:153:serveInstalled() [I] Run mode: prod 2023/07/18 13:20:38 cmd/web.go:154:serveInstalled() [I] Prepare to run web server 2023/07/18 13:20:38 routers/init.go:112:InitWebInstalled() [I] Git version: 2.40.1, Wire Protocol Version 2 Enabled (home: /data/home) 2023/07/18 13:20:39 ...les/setting/cache.go:75:loadCacheFrom() [I] Cache Service Enabled 2023/07/18 13:20:39 ...les/setting/cache.go:90:loadCacheFrom() [I] Last Commit Cache Service Enabled 2023/07/18 13:20:39 ...s/setting/session.go:74:loadSessionFrom() [I] Session Service Enabled 2023/07/18 13:20:39 ...s/storage/storage.go:176:initAttachments() [I] Initialising Attachment storage with type: minio 2023/07/18 13:20:39 ...les/storage/minio.go:81:NewMinioStorage() [I] Creating Minio storage at storage.googleapis.com:gitea-stg-aqrk with base path attachments/ 2023/07/18 13:20:40 routers/init.go:60:mustInit() [F] code.gitea.io/gitea/modules/storage.Init failed: permission denied ```
tobiasbp commented 2023-07-18 13:37:33 +00:00 (Migrated from gitea.com)

You need to remove all storage types from the sections and only leave the storage general section

Are you suggesting that inherited config from the previous version of Gitea should be deleted?

I have only configured the general storage section as seen below.

    storage:
      MINIO_ACCESS_KEY_ID: ***
      MINIO_BUCKET: my-existing-bucket
      MINIO_ENDPOINT: storage.googleapis.com
      MINIO_INSECURE_SKIP_VERIFY: false
      MINIO_SECRET_ACCESS_KEY: ***
      MINIO_USE_SSL: true
      SERVE_DIRECT: false
      STORAGE_TYPE: minio

UPDATE: Removed MINIO_LOCATION.

> You need to remove all storage types from the sections and only leave the storage general section Are you suggesting that inherited config from the previous version of Gitea should be deleted? I have only configured the general storage section as seen below. ``` storage: MINIO_ACCESS_KEY_ID: *** MINIO_BUCKET: my-existing-bucket MINIO_ENDPOINT: storage.googleapis.com MINIO_INSECURE_SKIP_VERIFY: false MINIO_SECRET_ACCESS_KEY: *** MINIO_USE_SSL: true SERVE_DIRECT: false STORAGE_TYPE: minio ``` UPDATE: Removed _MINIO_LOCATION_.
tobiasbp commented 2023-07-18 15:22:06 +00:00 (Migrated from gitea.com)

It looks to me like Gitea wants to re-create my existing bucket:

2023/07/18 15:18:13 ...les/storage/minio.go:81:NewMinioStorage() [I] Creating Minio storage at storage.googleapis.com:gitea-stg-aqrk with base path attachments/
2023/07/18 15:18:13 routers/init.go:60:mustInit() [F] code.gitea.io/gitea/modules/storage.Init failed: The requested bucket name is not available. The bucket namespace is shared by all users of the system. Please select a different name and try again.
It looks to me like _Gitea_ wants to re-create my existing bucket: ``` 2023/07/18 15:18:13 ...les/storage/minio.go:81:NewMinioStorage() [I] Creating Minio storage at storage.googleapis.com:gitea-stg-aqrk with base path attachments/ 2023/07/18 15:18:13 routers/init.go:60:mustInit() [F] code.gitea.io/gitea/modules/storage.Init failed: The requested bucket name is not available. The bucket namespace is shared by all users of the system. Please select a different name and try again. ```
tobiasbp commented 2023-07-18 16:36:59 +00:00 (Migrated from gitea.com)

Even if I set STORAGE_TYPE: local, Gitea still wants to create that bucket??

Config:

    storage:
      MINIO_ACCESS_KEY_ID: ***
      MINIO_BUCKET: my-existing-bucket
      MINIO_ENDPOINT: storage.googleapis.com
      MINIO_INSECURE_SKIP_VERIFY: false
      MINIO_SECRET_ACCESS_KEY: ***
      MINIO_USE_SSL: true
      SERVE_DIRECT: false
      STORAGE_TYPE: local 

Log:

2023/07/18 16:33:54 ...s/setting/session.go:74:loadSessionFrom() [I] Session Service Enabled
2023/07/18 16:33:54 ...s/storage/storage.go:176:initAttachments() [I] Initialising Attachment storage with type: minio
2023/07/18 16:33:54 ...les/storage/minio.go:81:NewMinioStorage() [I] Creating Minio storage at storage.googleapis.com:gitea-stg-aqrk with base path attachments/
2023/07/18 16:33:55 routers/init.go:60:mustInit() [F] code.gitea.io/gitea/modules/storage.Init failed: The requested bucket name is not available. The bucket namespace is shared by all users of the system. Please select a different name and try again
Even if I set _STORAGE_TYPE: local_, Gitea still wants to create that bucket?? Config: ``` storage: MINIO_ACCESS_KEY_ID: *** MINIO_BUCKET: my-existing-bucket MINIO_ENDPOINT: storage.googleapis.com MINIO_INSECURE_SKIP_VERIFY: false MINIO_SECRET_ACCESS_KEY: *** MINIO_USE_SSL: true SERVE_DIRECT: false STORAGE_TYPE: local ``` Log: ``` 2023/07/18 16:33:54 ...s/setting/session.go:74:loadSessionFrom() [I] Session Service Enabled 2023/07/18 16:33:54 ...s/storage/storage.go:176:initAttachments() [I] Initialising Attachment storage with type: minio 2023/07/18 16:33:54 ...les/storage/minio.go:81:NewMinioStorage() [I] Creating Minio storage at storage.googleapis.com:gitea-stg-aqrk with base path attachments/ 2023/07/18 16:33:55 routers/init.go:60:mustInit() [F] code.gitea.io/gitea/modules/storage.Init failed: The requested bucket name is not available. The bucket namespace is shared by all users of the system. Please select a different name and try again ```
pat-s commented 2023-07-18 17:32:47 +00:00 (Migrated from gitea.com)

On the first look this looks like an issue from Gitea upstream to me as we didn't do any changes on the storage section or related actions. In the end minio.go is executed with the values provided.

Assuming that you didn't yet do a successful 1.20 init + DB upgrade: does it work with the 1.19.4 image? This would proof that the chart is not the issue but that it's an upstream one.

On the first look this looks like an issue from Gitea upstream to me as we didn't do any changes on the `storage` section or related actions. In the end `minio.go` is executed with the values provided. Assuming that you didn't yet do a successful 1.20 init + DB upgrade: does it work with the 1.19.4 image? This would proof that the chart is not the issue but that it's an upstream one.
lunny commented 2023-07-19 04:32:34 +00:00 (Migrated from gitea.com)

Could you paste all your storage configurations? From v1.20, storage inherit becomes more strict.

Could you paste all your storage configurations? From v1.20, storage inherit becomes more strict.
tobiasbp commented 2023-07-19 07:39:02 +00:00 (Migrated from gitea.com)

Could you paste all your storage configurations? From v1.20, storage inherit becomes more strict.

These are the values I use with the v9.0.0 chart:

EDIT by pat-s: put into details block

helm -n git-backend get values gitea 

USER-SUPPLIED VALUES:
gitea:
  DOMAIN: ******
  ROOT_URL: ******
  admin:
    password: ******
  config:
    APP_NAME: Gitea (stg)
    RUN_MODE: dev
    admin:
      DEFAULT_EMAIL_NOTIFICATIONS: false
      DISABLE_REGULAR_ORG_CREATION: true
    cache:
      ADAPTER: redis
      ENABLED: true
      HOST: redis://redis-1.source-access.private:6379/0?pool_size=100&idle_timeout=180s
    database:
      DB_TYPE: postgres
      HOST: sql-proxy-gcloud-sqlproxy:5432
      NAME: gitea-db
      PASSWD: ********
      SSL_MODE: disable
      USER: gitea-user
    metrics:
      ENABLED: true
    oauth2_client:
      ENABLE_AUTO_REGISTRATION: true
      OPENID_CONNECT_SCOPES: openid profile email
      REGISTER_EMAIL_CONFIRM: false
      UPDATE_AVATAR: true
      USERNAME: email
    repository:
      DEFAULT_PRIVATE: true
      DISABLE_MIGRATIONS: true
      DISABLE_STARS: true
      DISABLED_REPO_UNITS: repo.wiki, repo.ext_wiki, repo.packages, repo.projects,
        repo.issues, repo.ext_issues, repo.pulls
      FORCE_PRIVATE: true
    security:
      DISABLE_WEBHOOKS: true
      INSTALL_LOCK: true
    server:
      DISABLE_SSH: true
      LANDING_PAGE: login
      LFS_START_SERVER: true
      PROTOCOL: http
    service:
      DEFAULT_ALLOW_CREATE_ORGANIZATION: false
      DEFAULT_ORG_VISIBILITY: private
      DEFAULT_USER_VISIBILITY: private
      DISABLE_REGISTRATION: true
      REQUIRE_SIGNIN_VIEW: true
      SHOW_MILESTONES_DASHBOARD_PAGE: false
      explore:
        DISABLE_USERS_PAGE: true
        REQUIRE_SIGNIN_VIEW: true
    storage:
      MINIO_ACCESS_KEY_ID: ******
      MINIO_BUCKET: gitea-stg-aqrk
      MINIO_ENDPOINT: storage.googleapis.com
      MINIO_INSECURE_SKIP_VERIFY: false
      MINIO_SECRET_ACCESS_KEY: ******
      MINIO_USE_SSL: true
      SERVE_DIRECT: false
      STORAGE_TYPE: minio
  log:
    LEVEL: debug
ingress:
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-production
  className: nginx
  enabled: true
  hosts:
  - host: ******
    paths:
    - path: /
      pathType: Prefix
  tls:
  - hosts:
    - ******
    secretName: git1-tls
memcached:
  enabled: false
persistence:
  claimName: data-gitea-0
  create: false
  enabled: true
  mount: true
  size: 20Gi
  storageClass: standard
postgresql:
  enabled: false
postgresql-ha:
  enabled: false
redis-cluster:
  enabled: false
resources:
  requests:
    cpu: 100m
    memory: 128Mi
strategy:
  type: Recreate
> Could you paste all your storage configurations? From v1.20, storage inherit becomes more strict. These are the values I use with the v9.0.0 chart: EDIT by pat-s: put into details block <details> ``` helm -n git-backend get values gitea USER-SUPPLIED VALUES: gitea: DOMAIN: ****** ROOT_URL: ****** admin: password: ****** config: APP_NAME: Gitea (stg) RUN_MODE: dev admin: DEFAULT_EMAIL_NOTIFICATIONS: false DISABLE_REGULAR_ORG_CREATION: true cache: ADAPTER: redis ENABLED: true HOST: redis://redis-1.source-access.private:6379/0?pool_size=100&idle_timeout=180s database: DB_TYPE: postgres HOST: sql-proxy-gcloud-sqlproxy:5432 NAME: gitea-db PASSWD: ******** SSL_MODE: disable USER: gitea-user metrics: ENABLED: true oauth2_client: ENABLE_AUTO_REGISTRATION: true OPENID_CONNECT_SCOPES: openid profile email REGISTER_EMAIL_CONFIRM: false UPDATE_AVATAR: true USERNAME: email repository: DEFAULT_PRIVATE: true DISABLE_MIGRATIONS: true DISABLE_STARS: true DISABLED_REPO_UNITS: repo.wiki, repo.ext_wiki, repo.packages, repo.projects, repo.issues, repo.ext_issues, repo.pulls FORCE_PRIVATE: true security: DISABLE_WEBHOOKS: true INSTALL_LOCK: true server: DISABLE_SSH: true LANDING_PAGE: login LFS_START_SERVER: true PROTOCOL: http service: DEFAULT_ALLOW_CREATE_ORGANIZATION: false DEFAULT_ORG_VISIBILITY: private DEFAULT_USER_VISIBILITY: private DISABLE_REGISTRATION: true REQUIRE_SIGNIN_VIEW: true SHOW_MILESTONES_DASHBOARD_PAGE: false explore: DISABLE_USERS_PAGE: true REQUIRE_SIGNIN_VIEW: true storage: MINIO_ACCESS_KEY_ID: ****** MINIO_BUCKET: gitea-stg-aqrk MINIO_ENDPOINT: storage.googleapis.com MINIO_INSECURE_SKIP_VERIFY: false MINIO_SECRET_ACCESS_KEY: ****** MINIO_USE_SSL: true SERVE_DIRECT: false STORAGE_TYPE: minio log: LEVEL: debug ingress: annotations: cert-manager.io/cluster-issuer: letsencrypt-production className: nginx enabled: true hosts: - host: ****** paths: - path: / pathType: Prefix tls: - hosts: - ****** secretName: git1-tls memcached: enabled: false persistence: claimName: data-gitea-0 create: false enabled: true mount: true size: 20Gi storageClass: standard postgresql: enabled: false postgresql-ha: enabled: false redis-cluster: enabled: false resources: requests: cpu: 100m memory: 128Mi strategy: type: Recreate ``` </details>
tobiasbp commented 2023-07-19 09:40:00 +00:00 (Migrated from gitea.com)

Assuming that you didn't yet do a successful 1.20 init + DB upgrade: does it work with the 1.19.4 image? This would proof that the chart is not the issue but that it's an upstream one.

I dumped my 1.20.0 database and reinstalled. Changing the Gitea version to 1.19.4 in chart 9.0.0 makes the minio storage work again:

image.tag: 1.19.4

Thus, the problem would seem to be unrelated to the chart.

I created an issue here: https://github.com/go-gitea/gitea/issues/25984

> Assuming that you didn't yet do a successful 1.20 init + DB upgrade: does it work with the 1.19.4 image? This would proof that the chart is not the issue but that it's an upstream one. I dumped my 1.20.0 database and reinstalled. Changing the Gitea version to 1.19.4 in chart 9.0.0 makes the _minio_ storage work again: ``` image.tag: 1.19.4 ``` Thus, the problem would seem to be unrelated to the chart. I created an issue here: https://github.com/go-gitea/gitea/issues/25984
pat-s commented 2023-07-19 10:01:45 +00:00 (Migrated from gitea.com)

Thanks for confirming!

Thanks for confirming!
Sign in to join this conversation.
No Milestone
No project
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: lunny/helm-chart#469
No description provided.