Gitea-HA multiple replica support #205

Closed

luhahn wants to merge 8 commits from gitea-ha into main

pull from: gitea-ha

merge into: lunny:main

Conversation 29 Commits 8 Files Changed 6 +270 -3

luhahn commented 2021-07-09 14:45:01 +00:00 (Migrated from gitea.com)

Copy Link

This PR provides support to run Gitea with multiple replicas out of the box.

You will need to:

• enable redis in the values
• update replicaCount > 1
• provide a readWriteMany PVC for gitea as existingVolume

memcached:
  enabled: false
  
redis:
  enabled: true
    
replicaCount: 2

gitea:
  persistence:
    enabled: true
    existingClaim: RWX-PVC-Gitea

This PR also adds postgresql-ha and redis-cluster as dependency to provide real HA support.

Fixes: #206

This PR provides support to run Gitea with multiple replicas out of the box. You will need to: - enable redis in the values - update replicaCount > 1 - provide a readWriteMany PVC for gitea as existingVolume ```yaml memcached: enabled: false redis: enabled: true replicaCount: 2 gitea: persistence: enabled: true existingClaim: RWX-PVC-Gitea ``` This PR also adds postgresql-ha and redis-cluster as dependency to provide real HA support. Fixes: #206

❤️ 5

luhahn commented 2021-07-12 10:46:46 +00:00 (Migrated from gitea.com)

Copy Link

Also moved the gitea.database.builtIn and gitea.cache.builtIn section into the dependencies directly

Also moved the gitea.database.builtIn and gitea.cache.builtIn section into the dependencies directly

viceice (Migrated from gitea.com) reviewed 2021-07-14 07:21:27 +00:00

viceice (Migrated from gitea.com) left a comment

Copy Link

What happens when memcached fails?

What happens when memcached fails?

templates/gitea/config.yaml

viceice (Migrated from gitea.com) commented 2021-07-14 07:21:27 +00:00

Copy Link

You can remove deprecated ISSUE_INDEXER_QUEUE_TYPE and ISSUE_INDEXER_QUEUE_CONN_STR. You anly need

{{- $_ := set .Values.gitea.config."queue.issue_indexer"  "TYPE" "redis" -}}

not sure how to properly quote queue.issue_indexer section.

I think you also need shared session configuration (redis or db)

{{- $_ := set .Values.gitea.config.session  "PROVIDER" "redis" -}}
{{- $_ := set .Values.gitea.config.session  "PROVIDER_CONFIG" (include "redis.dns" .) -}}

You can remove deprecated `ISSUE_INDEXER_QUEUE_TYPE` and `ISSUE_INDEXER_QUEUE_CONN_STR`. You anly need ``` {{- $_ := set .Values.gitea.config."queue.issue_indexer" "TYPE" "redis" -}} ``` not sure how to properly quote `queue.issue_indexer` section. I think you also need shared session configuration (redis or db) ``` {{- $_ := set .Values.gitea.config.session "PROVIDER" "redis" -}} {{- $_ := set .Values.gitea.config.session "PROVIDER_CONFIG" (include "redis.dns" .) -}} ```

viceice (Migrated from gitea.com) commented 2021-07-14 07:27:07 +00:00

Copy Link

Thank you for your Input :)

I've already played around with shared sessions, however I ran into some Issues and forgot about it :D

Will try again and adjust the rest.

Thank you for your Input :) I've already played around with shared sessions, however I ran into some Issues and forgot about it :D Will try again and adjust the rest.

viceice (Migrated from gitea.com) commented 2021-08-30 13:42:43 +00:00

Copy Link

Session now uses redis, had it configured with ingress previously.
Removed deprecated config for issue_indexer queue

Session now uses redis, had it configured with ingress previously. Removed deprecated config for issue_indexer queue

values.yaml

viceice (Migrated from gitea.com) commented 2021-07-14 07:23:25 +00:00

Copy Link

Share sessions between instances, see other comment

    session:
      PROVIDER: db

Share sessions between instances, see other comment ```yaml session: PROVIDER: db ```

luhahn commented 2021-07-14 07:49:37 +00:00 (Migrated from gitea.com)

Copy Link

Regarding memcached, when running HA I'd currently recommend redis instead of memcached.

Regarding memcached, when running HA I'd currently recommend redis instead of memcached.

viceice commented 2021-07-14 07:53:49 +00:00 (Migrated from gitea.com)

Copy Link

Regarding memcached, when running HA I'd currently recommend redis instead of memcached.

Thats what i'm currently do.

I also have a elastic search cluster for indexing.
Is it better to use db?

> Regarding memcached, when running HA I'd currently recommend redis instead of memcached. Thats what i'm currently do. I also have a elastic search cluster for indexing. Is it better to use `db`?

luhahn commented 2021-07-14 08:20:28 +00:00 (Migrated from gitea.com)

Copy Link

Regarding memcached, when running HA I'd currently recommend redis instead of memcached.

Thats what i'm currently do.

I also have a elastic search cluster for indexing.
Is it better to use db?

To be honest, I currently do not know which one will perform better. I plan to do some stress tests on the gitea cluster and document the results.

I choose db, because I already got postgresql-ha running, so it was the easiest to do :D

I'm thinking about adding elasticsearch as a dependency for indexing, but I'm worried, that we eventually will have too many dependencies.

> > Regarding memcached, when running HA I'd currently recommend redis instead of memcached. > > Thats what i'm currently do. > > I also have a elastic search cluster for indexing. > Is it better to use `db`? To be honest, I currently do not know which one will perform better. I plan to do some stress tests on the gitea cluster and document the results. I choose db, because I already got postgresql-ha running, so it was the easiest to do :D I'm thinking about adding elasticsearch as a dependency for indexing, but I'm worried, that we eventually will have too many dependencies.

luhahn commented 2021-08-30 12:52:41 +00:00 (Migrated from gitea.com)

Copy Link

hmm still having issues with the queue.issue_indexer :/

edit: ->solved :)

hmm still having issues with the queue.issue_indexer :/ edit: ->solved :)

luhahn commented 2021-08-30 13:44:20 +00:00 (Migrated from gitea.com)

Copy Link

This will be a breaking change, since i moved the following values

gitea.database.builtIn.mariadb.enabled -> mariadb.enabled
gitea.database.builtIn.postgresql.enabled -> postgresql.enabled
gitea.database.builtIn.mysql.enabled -> mysql.enabled
gitea.cache.builtIn.enabled -> memcached.enabled

This will be a breaking change, since i moved the following values gitea.database.builtIn.mariadb.enabled -> mariadb.enabled gitea.database.builtIn.postgresql.enabled -> postgresql.enabled gitea.database.builtIn.mysql.enabled -> mysql.enabled gitea.cache.builtIn.enabled -> memcached.enabled

luhahn commented 2021-08-30 13:45:44 +00:00 (Migrated from gitea.com)

Copy Link

This PR is ready if anyone wants to test it :)

This PR is ready if anyone wants to test it :)

zeripath (Migrated from gitea.com) approved these changes 2021-09-23 14:25:55 +00:00

pat-s commented 2021-09-24 20:10:57 +00:00 (Migrated from gitea.com)

Copy Link

What is the easiest way to test this build with terraform? I cannot find an easy way to use a helm chart from a branch.

Edit: testing with minikube is probably easiest. Added https://gitea.com/gitea/helm-chart/pulls/228 for instructions.

What is the easiest way to test this build with terraform? I cannot find an easy way to use a helm chart from a branch. Edit: testing with `minikube` is probably easiest. Added https://gitea.com/gitea/helm-chart/pulls/228 for instructions.

pat-s commented 2021-09-26 19:33:35 +00:00 (Migrated from gitea.com)

Copy Link

Please excuse for asking some possibly stupid questions and executing potentially questionable steps - relatively new to k8s replica deployments.

With this branch deployed, I see the following pods

default                gitea-0                                      1/1     Running   0               89s
default                gitea-memcached-584956987c-wdlsd             1/1     Running   0               9m16s
default                gitea-postgresql-0                           1/1     Running   0               9m16s
default                gitea-redis-master-0                         1/1     Running   0               9m16s
default                gitea-redis-replicas-0                       1/1     Running   1 (8m34s ago)   9m16s
default                gitea-redis-replicas-1                       1/1     Running   0               8m29s

I see replicas for redis but not for gitea - is gitea-memcached the second one here?
Also I see no replicaton for the postgres pod - is this expected?

Next, for gitea-redis-replicas-0 I saw the following during startup

  Normal   Started           11m (x2 over 11m)  kubelet            Started container redis
  Warning  Unhealthy         11m (x6 over 11m)  kubelet            Readiness probe failed:
Could not connect to Redis at gitea-redis-master-0.gitea-redis-headless.default.svc.cluster.local:6379: Name or service not known
  Warning  Unhealthy  11m (x5 over 11m)  kubelet  Liveness probe failed:
Could not connect to Redis at gitea-redis-master-0.gitea-redis-headless.default.svc.cluster.local:6379: Name or service not known
  Normal  Killing  11m  kubelet  Container redis failed liveness probe, will be restarted
  Normal  Pulled   11m  kubelet  Container image "docker.io/bitnami/redis:6.2.4-debian-10-r13" already present on machine

How could I effectively test HA works? I deleted the gitea-0 pod and lost the connection as port 3000 was gone (not sure if this a good test since I used kubectl port-forward).

Please excuse for asking some possibly stupid questions and executing potentially questionable steps - relatively new to k8s replica deployments. With this branch deployed, I see the following pods ``` default gitea-0 1/1 Running 0 89s default gitea-memcached-584956987c-wdlsd 1/1 Running 0 9m16s default gitea-postgresql-0 1/1 Running 0 9m16s default gitea-redis-master-0 1/1 Running 0 9m16s default gitea-redis-replicas-0 1/1 Running 1 (8m34s ago) 9m16s default gitea-redis-replicas-1 1/1 Running 0 8m29s ``` I see replicas for redis but not for gitea - is `gitea-memcached` the second one here? Also I see no replicaton for the postgres pod - is this expected? Next, for `gitea-redis-replicas-0` I saw the following during startup ``` Normal Started 11m (x2 over 11m) kubelet Started container redis Warning Unhealthy 11m (x6 over 11m) kubelet Readiness probe failed: Could not connect to Redis at gitea-redis-master-0.gitea-redis-headless.default.svc.cluster.local:6379: Name or service not known Warning Unhealthy 11m (x5 over 11m) kubelet Liveness probe failed: Could not connect to Redis at gitea-redis-master-0.gitea-redis-headless.default.svc.cluster.local:6379: Name or service not known Normal Killing 11m kubelet Container redis failed liveness probe, will be restarted Normal Pulled 11m kubelet Container image "docker.io/bitnami/redis:6.2.4-debian-10-r13" already present on machine ``` How could I effectively test HA works? I deleted the `gitea-0` pod and lost the connection as port 3000 was gone (not sure if this a good test since I used `kubectl port-forward`).

luhahn commented 2021-09-27 14:24:21 +00:00 (Migrated from gitea.com)

Copy Link

Please excuse for asking some possibly stupid questions and executing potentially questionable steps - relatively new to k8s replica deployments.

With this branch deployed, I see the following pods

default                gitea-0                                      1/1     Running   0               89s
default                gitea-memcached-584956987c-wdlsd             1/1     Running   0               9m16s
default                gitea-postgresql-0                           1/1     Running   0               9m16s
default                gitea-redis-master-0                         1/1     Running   0               9m16s
default                gitea-redis-replicas-0                       1/1     Running   1 (8m34s ago)   9m16s
default                gitea-redis-replicas-1                       1/1     Running   0               8m29s

I see replicas for redis but not for gitea - is gitea-memcached the second one here?
Also I see no replicaton for the postgres pod - is this expected?

Next, for gitea-redis-replicas-0 I saw the following during startup

  Normal   Started           11m (x2 over 11m)  kubelet            Started container redis
  Warning  Unhealthy         11m (x6 over 11m)  kubelet            Readiness probe failed:
Could not connect to Redis at gitea-redis-master-0.gitea-redis-headless.default.svc.cluster.local:6379: Name or service not known
  Warning  Unhealthy  11m (x5 over 11m)  kubelet  Liveness probe failed:
Could not connect to Redis at gitea-redis-master-0.gitea-redis-headless.default.svc.cluster.local:6379: Name or service not known
  Normal  Killing  11m  kubelet  Container redis failed liveness probe, will be restarted
  Normal  Pulled   11m  kubelet  Container image "docker.io/bitnami/redis:6.2.4-debian-10-r13" already present on machine

How could I effectively test HA works? I deleted the gitea-0 pod and lost the connection as port 3000 was gone (not sure if this a good test since I used kubectl port-forward).

Sorry, I forgot to mention this i the initial message. Youll also need to disable memcached if you use redis. I should probably add a check here, if either one of them is enabled.

Have you set replicaCount > 1 for gitea?

> Please excuse for asking some possibly stupid questions and executing potentially questionable steps - relatively new to k8s replica deployments. > > With this branch deployed, I see the following pods > > ``` > default gitea-0 1/1 Running 0 89s > default gitea-memcached-584956987c-wdlsd 1/1 Running 0 9m16s > default gitea-postgresql-0 1/1 Running 0 9m16s > default gitea-redis-master-0 1/1 Running 0 9m16s > default gitea-redis-replicas-0 1/1 Running 1 (8m34s ago) 9m16s > default gitea-redis-replicas-1 1/1 Running 0 8m29s > > ``` > > I see replicas for redis but not for gitea - is `gitea-memcached` the second one here? > Also I see no replicaton for the postgres pod - is this expected? > > Next, for `gitea-redis-replicas-0` I saw the following during startup > > ``` > Normal Started 11m (x2 over 11m) kubelet Started container redis > Warning Unhealthy 11m (x6 over 11m) kubelet Readiness probe failed: > Could not connect to Redis at gitea-redis-master-0.gitea-redis-headless.default.svc.cluster.local:6379: Name or service not known > Warning Unhealthy 11m (x5 over 11m) kubelet Liveness probe failed: > Could not connect to Redis at gitea-redis-master-0.gitea-redis-headless.default.svc.cluster.local:6379: Name or service not known > Normal Killing 11m kubelet Container redis failed liveness probe, will be restarted > Normal Pulled 11m kubelet Container image "docker.io/bitnami/redis:6.2.4-debian-10-r13" already present on machine > ``` > > How could I effectively test HA works? I deleted the `gitea-0` pod and lost the connection as port 3000 was gone (not sure if this a good test since I used `kubectl port-forward`). Sorry, I forgot to mention this i the initial message. Youll also need to disable memcached if you use redis. I should probably add a check here, if either one of them is enabled. Have you set replicaCount > 1 for gitea?

lunny commented 2021-10-20 01:58:29 +00:00 (Migrated from gitea.com)

Copy Link

I think there are still some configurations absent include queue. I list them below:

• Sessions: the values could be "memory", "file", "redis", "mysql", "postgres", "couchbase", "memcache", "db". And the default value is memory, I recommand to use db or redis. If use redis, PROVIDER_CONFIG is needed.
• Cache: the values could be "memory", "redis", "memcache", default cache is memory. And for ha mode, redis or memcache is suitable.
• Indexes: issues indexer supports bleve, db or elasticsearch, the default is bleve. It should be changed to db or elasticsearch.
• code indexer supports bleve or elasticsearch, the default is bleve. It should be elasticsearch for HA mode.
• Storages: storages for attachments, avatars, lfs, archives and so on. The default is local file system. It's better to store them to a minio instance.

I think there are still some configurations absent include `queue`. I list them below: - [x] Sessions: the values could be `"memory", "file", "redis", "mysql", "postgres", "couchbase", "memcache", "db"`. And the default value is `memory`, I recommand to use `db` or `redis`. If use `redis`, `PROVIDER_CONFIG` is needed. - [x] Cache: the values could be `"memory", "redis", "memcache"`, default cache is `memory`. And for ha mode, `redis` or `memcache` is suitable. - [x] Indexes: issues indexer supports `bleve, db or elasticsearch`, the default is `bleve`. It should be changed to `db` or `elasticsearch`. - [x] code indexer supports `bleve` or `elasticsearch`, the default is `bleve`. It should be `elasticsearch` for HA mode. - [x] Storages: storages for `attachments`, `avatars`, `lfs`, `archives` and so on. The default is local file system. It's better to store them to a `minio` instance.

👍 1

luhahn commented 2021-11-09 10:02:24 +00:00 (Migrated from gitea.com)

Copy Link

@lunny Sorry, it took me quite a while to recover and get back to work. However

Session:
This is already included, I set redis as default including provider config, if
replicaCount > 1 and (recently added) if no session config was provided by the user.

    {{- /* multiple replicas */ -}}
    {{- if gt .Values.replicaCount 1.0 -}}
    {{- if not (hasKey .Values.gitea.config.session "PROVIDER") -}}
    {{- $_ := set .Values.gitea.config.session  "PROVIDER" "redis" -}}
    {{- end -}}
    {{- if not (hasKey .Values.gitea.config.session "PROVIDER_CONFIG") -}}
    {{- $_ := set .Values.gitea.config.session  "PROVIDER_CONFIG" (include "redis.dns" .) -}}
    {{- end -}}
    {{- end -}}

Cache:
Also already taken care of automatically, if no user config is provided for the cache:

    {{- /* cache default settings */ -}}
    {{- if or .Values.memcached.enabled (index .Values "redis-cluster").enabled .Values.redis.enabled -}}
    {{- $_ := set .Values.gitea.config.cache "ENABLED" "true" -}}
    {{- $_ := set .Values.gitea.config.cache "ADAPTER" (ternary "memcache" "redis" .Values.memcached.enabled) -}}
    {{- if not (.Values.gitea.config.cache.HOST) -}}
    {{- $_ := set .Values.gitea.config.cache "HOST" (ternary (include "memcached.dns" .) (include "redis.dns" .) .Values.memcached.enabled) -}}
    {{- end -}}
    {{- end -}}

Indexer:
Same as cache

{{- $_ := set .Values.gitea.config.indexer "ISSUE_INDEXER_TYPE" "db" -}}

I will check on the storage part :)

I will also need to check some parts of the config generation.

@lunny Sorry, it took me quite a while to recover and get back to work. However Session: This is already included, I set redis as default including provider config, if replicaCount > 1 and (recently added) if no session config was provided by the user. ```yaml {{- /* multiple replicas */ -}} {{- if gt .Values.replicaCount 1.0 -}} {{- if not (hasKey .Values.gitea.config.session "PROVIDER") -}} {{- $_ := set .Values.gitea.config.session "PROVIDER" "redis" -}} {{- end -}} {{- if not (hasKey .Values.gitea.config.session "PROVIDER_CONFIG") -}} {{- $_ := set .Values.gitea.config.session "PROVIDER_CONFIG" (include "redis.dns" .) -}} {{- end -}} {{- end -}} ``` Cache: Also already taken care of automatically, if no user config is provided for the cache: ```yaml {{- /* cache default settings */ -}} {{- if or .Values.memcached.enabled (index .Values "redis-cluster").enabled .Values.redis.enabled -}} {{- $_ := set .Values.gitea.config.cache "ENABLED" "true" -}} {{- $_ := set .Values.gitea.config.cache "ADAPTER" (ternary "memcache" "redis" .Values.memcached.enabled) -}} {{- if not (.Values.gitea.config.cache.HOST) -}} {{- $_ := set .Values.gitea.config.cache "HOST" (ternary (include "memcached.dns" .) (include "redis.dns" .) .Values.memcached.enabled) -}} {{- end -}} {{- end -}} ``` Indexer: Same as cache ```yaml {{- $_ := set .Values.gitea.config.indexer "ISSUE_INDEXER_TYPE" "db" -}} ``` I will check on the storage part :) I will also need to check some parts of the config generation.

lunny commented 2021-11-09 10:08:46 +00:00 (Migrated from gitea.com)

Copy Link

@lunny Sorry, it took me quite a while to recover and get back to work. However

Session:
This is already included, I set redis as default including provider config, if
replicaCount > 1 and (recently added) if no session config was provided by the user.

    {{- /* multiple replicas */ -}}
    {{- if gt .Values.replicaCount 1.0 -}}
    {{- if not (hasKey .Values.gitea.config.session "PROVIDER") -}}
    {{- $_ := set .Values.gitea.config.session  "PROVIDER" "redis" -}}
    {{- end -}}
    {{- if not (hasKey .Values.gitea.config.session "PROVIDER_CONFIG") -}}
    {{- $_ := set .Values.gitea.config.session  "PROVIDER_CONFIG" (include "redis.dns" .) -}}
    {{- end -}}
    {{- end -}}

Cache:
Also already taken care of automatically, if no user config is provided for the cache:

    {{- /* cache default settings */ -}}
    {{- if or .Values.memcached.enabled (index .Values "redis-cluster").enabled .Values.redis.enabled -}}
    {{- $_ := set .Values.gitea.config.cache "ENABLED" "true" -}}
    {{- $_ := set .Values.gitea.config.cache "ADAPTER" (ternary "memcache" "redis" .Values.memcached.enabled) -}}
    {{- if not (.Values.gitea.config.cache.HOST) -}}
    {{- $_ := set .Values.gitea.config.cache "HOST" (ternary (include "memcached.dns" .) (include "redis.dns" .) .Values.memcached.enabled) -}}
    {{- end -}}
    {{- end -}}

Indexer:
Same as cache

{{- $_ := set .Values.gitea.config.indexer "ISSUE_INDEXER_TYPE" "db" -}}

I will check on the storage part :)

I will also need to check some parts of the config generation.

For indexers, there are issue indexer and code indexer. I think maybe you missed the last one.

> @lunny Sorry, it took me quite a while to recover and get back to work. However > > Session: > This is already included, I set redis as default including provider config, if > replicaCount > 1 and (recently added) if no session config was provided by the user. > > ```yaml > {{- /* multiple replicas */ -}} > {{- if gt .Values.replicaCount 1.0 -}} > {{- if not (hasKey .Values.gitea.config.session "PROVIDER") -}} > {{- $_ := set .Values.gitea.config.session "PROVIDER" "redis" -}} > {{- end -}} > {{- if not (hasKey .Values.gitea.config.session "PROVIDER_CONFIG") -}} > {{- $_ := set .Values.gitea.config.session "PROVIDER_CONFIG" (include "redis.dns" .) -}} > {{- end -}} > {{- end -}} > ``` > > Cache: > Also already taken care of automatically, if no user config is provided for the cache: > > ```yaml > {{- /* cache default settings */ -}} > {{- if or .Values.memcached.enabled (index .Values "redis-cluster").enabled .Values.redis.enabled -}} > {{- $_ := set .Values.gitea.config.cache "ENABLED" "true" -}} > {{- $_ := set .Values.gitea.config.cache "ADAPTER" (ternary "memcache" "redis" .Values.memcached.enabled) -}} > {{- if not (.Values.gitea.config.cache.HOST) -}} > {{- $_ := set .Values.gitea.config.cache "HOST" (ternary (include "memcached.dns" .) (include "redis.dns" .) .Values.memcached.enabled) -}} > {{- end -}} > {{- end -}} > ``` > > Indexer: > Same as cache > > ```yaml > {{- $_ := set .Values.gitea.config.indexer "ISSUE_INDEXER_TYPE" "db" -}} > ``` > > I will check on the storage part :) > > I will also need to check some parts of the config generation. For indexers, there are issue indexer and code indexer. I think maybe you missed the last one.

luhahn commented 2021-11-11 09:10:04 +00:00 (Migrated from gitea.com)

Copy Link

@lunny regarding the storage. I think it is nice to have a minio instance for storage. However we already have a lot of dependencies in the chart, like postgresql, redis, mysql, mariadb, etc. I fear, that if we keep adding dependencies the chart will get too complicated to use. For example, the Gitlab Helm Chart also has lots and lots of dependencies and it is really horrific to use.

HA already works fine when using a RWX PVC, I think it might be better to not directly include minio.

Not blocking the Idea of minio as a dependency, but I want to discuss this :)

Same goes for the code indexer.

If I understood correctly, the code indexing feature is optional.

REPO_INDEXER_ENABLED -> false by default

I wouldn't want to add elasticsearch as a dependency, same reason as stated above :)
We should add a description which makes clear, that if HA is used, the REPO_INDEXER should be set to elasticsearch.

@lunny regarding the storage. I think it is nice to have a minio instance for storage. However we already have a lot of dependencies in the chart, like postgresql, redis, mysql, mariadb, etc. I fear, that if we keep adding dependencies the chart will get too complicated to use. For example, the Gitlab Helm Chart also has lots and lots of dependencies and it is really horrific to use. HA already works fine when using a RWX PVC, I think it might be better to not directly include minio. Not blocking the Idea of minio as a dependency, but I want to discuss this :) Same goes for the code indexer. If I understood correctly, the code indexing feature is optional. REPO_INDEXER_ENABLED -> false by default I wouldn't want to add elasticsearch as a dependency, same reason as stated above :) We should add a description which makes clear, that if HA is used, the REPO_INDEXER should be set to elasticsearch.

❤️ 1

lunny commented 2021-11-11 10:07:04 +00:00 (Migrated from gitea.com)

Copy Link

@lunny regarding the storage. I think it is nice to have a minio instance for storage. However we already have a lot of dependencies in the chart, like postgresql, redis, mysql, mariadb, etc. I fear, that if we keep adding dependencies the chart will get too complicated to use. For example, the Gitlab Helm Chart also has lots and lots of dependencies and it is really horrific to use.

HA already works fine when using a RWX PVC, I think it might be better to not directly include minio.

Not blocking the Idea of minio as a dependency, but I want to discuss this :)

Same goes for the code indexer.

If I understood correctly, the code indexing feature is optional.

REPO_INDEXER_ENABLED -> false by default

I wouldn't want to add elasticsearch as a dependency, same reason as stated above :)
We should add a description which makes clear, that if HA is used, the REPO_INDEXER should be set to elasticsearch.

But if code indexer set to true, and it's a default value bleve. The second gitea instance maybe hang.

For storage, if it's a RWX PVC, That's no problem.

> @lunny regarding the storage. I think it is nice to have a minio instance for storage. However we already have a lot of dependencies in the chart, like postgresql, redis, mysql, mariadb, etc. I fear, that if we keep adding dependencies the chart will get too complicated to use. For example, the Gitlab Helm Chart also has lots and lots of dependencies and it is really horrific to use. > > HA already works fine when using a RWX PVC, I think it might be better to not directly include minio. > > Not blocking the Idea of minio as a dependency, but I want to discuss this :) > > Same goes for the code indexer. > > If I understood correctly, the code indexing feature is optional. > > REPO_INDEXER_ENABLED -> false by default > > I wouldn't want to add elasticsearch as a dependency, same reason as stated above :) > We should add a description which makes clear, that if HA is used, the REPO_INDEXER should be set to elasticsearch. > But if code indexer set to true, and it's a default value `bleve`. The second gitea instance maybe hang. For storage, if it's a RWX PVC, That's no problem.

justusbunsi commented 2021-12-18 11:33:23 +00:00 (Migrated from gitea.com)

Copy Link

I'm wondering if Gitea could handle PodDisruptionBudgets as to replace one pod at a time to keep Gitea up?
Is this too risky from application perspective regarding data(base) consistency?

I'm wondering if Gitea could handle `PodDisruptionBudgets` as to _replace one pod at a time to keep Gitea up_? Is this too risky from application perspective regarding data(base) consistency?

pat-s commented 2022-03-26 16:43:33 +00:00 (Migrated from gitea.com)

Copy Link

In case some testing/help is needed, LMK :)

In case some testing/help is needed, LMK :)

pat-s commented 2022-05-17 18:41:07 +00:00 (Migrated from gitea.com)

Copy Link

@luhahn @justusbunsi

I'd be interesting in testing/pushing this forward. I could start by merging master first and resolving conflicts, then continue with some testing/feedback.

@luhahn @justusbunsi I'd be interesting in testing/pushing this forward. I could start by merging `master` first and resolving conflicts, then continue with some testing/feedback.

👍 1

luhahn commented 2022-05-18 09:08:15 +00:00 (Migrated from gitea.com)

Copy Link

I will hopefully continue next week, the last time i checked, i had some issues.

I will hopefully continue next week, the last time i checked, i had some issues.

justusbunsi commented 2022-06-11 11:50:10 +00:00 (Migrated from gitea.com)

Copy Link

As per discord discussion, running multiple Gitea instances on the same database and storage requires some sort of internal leader election. Without the instances being aware of each others everyone would execute cron tasks. This can lead to unexpected inconsistencies.

As per discord discussion, running multiple Gitea instances on the same database and storage requires some sort of internal leader election. Without the instances being aware of each others everyone would execute cron tasks. This can lead to unexpected inconsistencies.

justusbunsi commented 2022-06-26 12:58:22 +00:00 (Migrated from gitea.com)

Copy Link

Just realized that when running multiple replicas, all init containers would run x times. But this might be a problem. Think of upgrading Gitea. One of the init container executes a gitea migrate. We wouldn't want to have this action done twice or in parallel.

#332

Just realized that when running multiple replicas, all init containers would run x times. But this might be a problem. Think of upgrading Gitea. One of the init container executes a `gitea migrate`. We wouldn't want to have this action done twice or in parallel. #332

viceice commented 2022-08-24 13:40:56 +00:00 (Migrated from gitea.com)

Copy Link

As per discord discussion, running multiple Gitea instances on the same database and storage requires some sort of internal leader election. Without the instances being aware of each others everyone would execute cron tasks. This can lead to unexpected inconsistencies.

I maybe see those issues sometimes now.
symtoms:

• push a commit, which isn't visible to ci job yet
• broken PR, which never gets it's checks ready

> As per discord discussion, running multiple Gitea instances on the same database and storage requires some sort of internal leader election. Without the instances being aware of each others everyone would execute cron tasks. This can lead to unexpected inconsistencies. I maybe see those issues sometimes now. symtoms: - push a commit, which isn't visible to ci job yet - broken PR, which never gets it's checks ready

😕 1

lunny commented 2023-03-07 13:50:34 +00:00 (Migrated from gitea.com)

Copy Link

Since now we have an option to disable migration when start, maybe we can let one instance to run migration but others not?

Since now we have an option to disable migration when start, maybe we can let one instance to run migration but others not?

justusbunsi commented 2023-03-07 15:49:35 +00:00 (Migrated from gitea.com)

Copy Link

Since now we have an option to disable migration when start, maybe we can let one instance to run migration but others not?

That could actually be a way into the right direction. Can you share the PR reference that implemented this?

I think it is important for using that functionality, to move away from the complex structure of the init containers. We cannot tell which replica shall act as leader and run the migration. All replicas are identical regarding configuration.

app.ini creation and other preparation must be done once for all replicas. They share those files.

Another problem is that we cannot start a new Gitea version along with the old one to perform a rollover. I currently see no way to achieve it without a downtime. The database must be migrated in order to run the newer Gitea version which would break older running versions. Migrate is required for major releases afaik, maybe for patch release too.

An option would be to first scale down all non-leader replicas, then replace the leader with the new version, let it handle the migration, scale non-leader replicas up with new Gitea version. To achieve that, we have to know which instance is the leader instance.

> Since now we have an option to disable migration when start, maybe we can let one instance to run migration but others not? That could actually be a way into the right direction. Can you share the PR reference that implemented this? I think it is important for using that functionality, to move away from the complex structure of the init containers. We cannot tell which replica shall act as leader and run the migration. All replicas are identical regarding configuration. app.ini creation and other preparation must be done once for all replicas. They share those files. Another problem is that we cannot start a new Gitea version along with the old one to perform a rollover. I currently see no way to achieve it without a downtime. The database must be migrated in order to run the newer Gitea version which would break older running versions. Migrate is required for major releases afaik, maybe for patch release too. An option would be to first scale down all non-leader replicas, then replace the leader with the new version, let it handle the migration, scale non-leader replicas up with new Gitea version. To achieve that, we have to know which instance is the leader instance.

lunny commented 2023-03-07 15:59:50 +00:00 (Migrated from gitea.com)

Copy Link

Since now we have an option to disable migration when start, maybe we can let one instance to run migration but others not?

That could actually be a way into the right direction. Can you share the PR reference that implemented this?

I think it is important for using that functionality, to move away from the complex structure of the init containers. We cannot tell which replica shall act as leader and run the migration. All replicas are identical regarding configuration.

app.ini creation and other preparation must be done once for all replicas. They share those files.

Another problem is that we cannot start a new Gitea version along with the old one to perform a rollover. I currently see no way to achieve it without a downtime. The database must be migrated in order to run the newer Gitea version which would break older running versions. Migrate is required for major releases afaik, maybe for patch release too.

An option would be to first scale down all non-leader replicas, then replace the leader with the new version, let it handle the migration, scale non-leader replicas up with new Gitea version. To achieve that, we have to know which instance is the leader instance.

It's https://github.com/go-gitea/gitea/pull/22053 . I also think maybe it's the only way to upgrade.

> > Since now we have an option to disable migration when start, maybe we can let one instance to run migration but others not? > > That could actually be a way into the right direction. Can you share the PR reference that implemented this? > > I think it is important for using that functionality, to move away from the complex structure of the init containers. We cannot tell which replica shall act as leader and run the migration. All replicas are identical regarding configuration. > > app.ini creation and other preparation must be done once for all replicas. They share those files. > > Another problem is that we cannot start a new Gitea version along with the old one to perform a rollover. I currently see no way to achieve it without a downtime. The database must be migrated in order to run the newer Gitea version which would break older running versions. Migrate is required for major releases afaik, maybe for patch release too. > > An option would be to first scale down all non-leader replicas, then replace the leader with the new version, let it handle the migration, scale non-leader replicas up with new Gitea version. To achieve that, we have to know which instance is the leader instance. It's https://github.com/go-gitea/gitea/pull/22053 . I also think maybe it's the only way to upgrade.

👍 1

pat-s commented 2023-04-01 10:34:42 +00:00 (Migrated from gitea.com)

Copy Link

Re:memcached

The bitnami chart says one can set architecture: "high-availability" and everything should be handled behind the scences.

Re:memcached The bitnami chart says one can set `architecture: "high-availability"` and everything should be handled behind the scences.

pat-s commented 2023-04-01 10:40:44 +00:00 (Migrated from gitea.com)

Copy Link

That could bring everything down to the following (aiming for minimal changes to the helm chart):

• Deploy/configure built-in memcached in HA mode
• Disable repo-indexing for bleve or use elastisearch or similar
• Use a RWX for persistent storage
• Use an external DB which is HA-capable (e.g. RDS on AWS)

In addition we could try to use a deployment instead of a statefulset for the gitea pods to avoid creating replicated PVs for each replica (see discussion in #426).

I will go ahead and create a new branch with some required changes and test the above config in a toy cluster.

That could bring everything down to the following (aiming for minimal changes to the helm chart): - Deploy/configure built-in `memcached` in HA mode - Disable repo-indexing for `bleve` or use `elastisearch` or similar - Use a RWX for persistent storage - Use an external DB which is HA-capable (e.g. RDS on AWS) In addition we could try to use a deployment instead of a statefulset for the gitea pods to avoid creating replicated PVs for each replica (see discussion in #426). I will go ahead and create a new branch with some required changes and test the above config in a toy cluster.

pat-s commented 2023-04-02 08:15:21 +00:00 (Migrated from gitea.com)

Copy Link

The bitnami chart says one can set architecture: "high-availability" and everything should be handled behind the scences.

Tried it, doesn't work. Doesn't matter it memcached itself is running in HA, the operations in the back don't work without issues.

> The bitnami chart says one can set architecture: "high-availability" and everything should be handled behind the scences. Tried it, doesn't work. Doesn't matter it `memcached` itself is running in HA, the operations in the back don't work without issues.

pat-s commented 2023-04-14 10:11:30 +00:00 (Migrated from gitea.com)

Copy Link

I'll close here in favor of #437 which has almost everything of this PR + more.

I'll close here in favor of #437 which has almost everything of this PR + more.

Pull request closed

Please reopen this pull request to perform a merge.

Sign in to join this conversation.

Reviewers

No Reviewers

zeripath

Labels

Clear labels

has/backport

in progress

invalid

kind/breaking

kind/bug

kind/build

kind/dependency

kind/deployment

kind/docs

kind/enhancement

kind/feature

kind/lint

kind/proposal

kind/question

kind/refactor

kind/security

kind/testing

kind/translation

kind/ui

need/backport

priority/critical

priority/low

priority/maybe

priority/medium

reviewed/duplicate

reviewed/invalid

reviewed/wontfix

skip-changelog

status/blocked

status/needs-feedback

status/needs-reviews

status/wip

upstream/gitea

upstream/other

No Label

kind/breaking

kind/feature

Milestone

No items

No Milestone

Projects

Clear projects

No project

Assignees

Clear assignees

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: lunny/helm-chart#205

No description provided.