lunny/helm-chart
1
0
Fork 1
Code Issues36 Pull Requests14 Actions Packages Projects Releases93 Wiki Activity

Add documentation for HA setup #350

Closed
pat-s wants to merge 8 commits from refs/pull/350/head into main
Conversation 9 Commits 8 Files Changed 2 +1195 -5359
pat-s commented 2022-08-21 12:13:32 +00:00 (Migrated from gitea.com)

This should serve as an initial draft to collect information about HA.

I am not bound to the type or location of the information - feel free to move.
I thought putting it into the README will make the README too long - having a dedicated document for this topic seemed reasonable to me.

This content is mainly based on and , especially the information by @lunny
and @viceice.

@lunny @viceice @justusbunsi @luhahn
Feel free to edit/propose changes, I am sure that not everything might be on point or I missed something.

WRT to listing the non-supported indexers: I thought it might be interesting to some people and encourage them eventually to add support - after realizing that

This should serve as an initial draft to collect information about HA. I am not bound to the type or location of the information - feel free to move. I thought putting it into the README will make the README too long - having a dedicated document for this topic seemed reasonable to me. This content is mainly based on #205 and #206, especially the information by @lunny and @viceice. @lunny @viceice @justusbunsi @luhahn Feel free to edit/propose changes, I am sure that not everything might be on point or I missed something. WRT to listing the non-supported indexers: I thought it might be interesting to some people and encourage them eventually to add support - after realizing that
👍 1 ❤️ 1
solacelost commented 2022-09-10 03:38:32 +00:00 (Migrated from gitea.com)

Any chance we could provide a toggle in values.yml to move the template from a StatefulSet to a Deployment? I'm willing to contribute the work there, but having multiple RWX PVCs (as in a multi-replica StatefulSet) != having a shared RWX PVC.

Also happy to contribute a bit to this guide, like updating some of the values.yml recommendations (builtIn is gone, adding in storage options like Rook for object and RWX storage, etc.). I think it might be nice to link to the existing.

I would also like to see the aspirational indexers linked to issues proposing their support (existing already or otherwise).

Any chance we could provide a toggle in values.yml to move the template from a StatefulSet to a Deployment? I'm willing to contribute the work there, but having multiple RWX PVCs (as in a multi-replica StatefulSet) != having a shared RWX PVC. Also happy to contribute a bit to this guide, like updating some of the values.yml recommendations (`builtIn` is gone, adding in storage options like Rook for object and RWX storage, etc.). I think it might be nice to link to the existing. I would also like to see the aspirational indexers linked to issues proposing their support (existing already or otherwise).
pat-s commented 2022-09-14 08:45:38 +00:00 (Migrated from gitea.com)

Any chance we could provide a toggle in values.yml to move the template from a StatefulSet to a Deployment? I'm willing to contribute the work there, but having multiple RWX PVCs (as in a multi-replica StatefulSet) != having a shared RWX PVC.

What would be the advantage here? Or in other words, which current limitation would this solve?

> Any chance we could provide a toggle in values.yml to move the template from a StatefulSet to a Deployment? I'm willing to contribute the work there, but having multiple RWX PVCs (as in a multi-replica StatefulSet) != having a shared RWX PVC. What would be the advantage here? Or in other words, which current limitation would this solve?
solacelost commented 2022-09-14 13:36:07 +00:00 (Migrated from gitea.com)

Any chance we could provide a toggle in values.yml to move the template from a StatefulSet to a Deployment? I'm willing to contribute the work there, but having multiple RWX PVCs (as in a multi-replica StatefulSet) != having a shared RWX PVC.

What would be the advantage here? Or in other words, which current limitation would this solve?

In a StatefulSet, scaling the number of replicas scales the number of PVCs to align with the number of pods. That is, each pod in a StatefulSet replica gets a unique PVC (at least when configured with a volumeClaimTemplate, as in the case of the Gitea StatefulSet from this chart).

Moving Gitea backend services to those that scale easily is great, but moving to an RWX PVC for Gitea's repositories doesn't automatically allow us to scale the StatefulSet to access those repositories from multiple load-balanced instances. It would simple create a new PVC for the new replica, and you would have different copies of the /data-mounted PV, each with different data.

StatefulSet is a great mechanism for scaling RWO storage PVCs alongside replicas of an application that handles clustering natively, often requiring some amount of consistent identity between instances of pods (where "0" might be a leader, and any replica higher than "0" might be a follower, but "0" can be replaced and remain as "0").

In the case of Gitea replicas, the configuration you're documenting is attempting to offload most of the functionality to external HA components (DB, queue, cache, blob store, etc.) and keep the basic Git repositories (stored in /data/git/gitea-repositories) shared between multiple replicas. That will not work with a StatefulSet, even with RWX storage.

> > Any chance we could provide a toggle in values.yml to move the template from a StatefulSet to a Deployment? I'm willing to contribute the work there, but having multiple RWX PVCs (as in a multi-replica StatefulSet) != having a shared RWX PVC. > > What would be the advantage here? Or in other words, which current limitation would this solve? In a StatefulSet, scaling the number of replicas scales the number of PVCs to align with the number of pods. That is, each pod in a StatefulSet replica gets a unique PVC (at least when configured with a volumeClaimTemplate, as in the case of the Gitea StatefulSet from this chart). Moving Gitea backend services to those that scale easily is great, but moving to an RWX PVC for Gitea's repositories doesn't automatically allow us to scale the StatefulSet to access those repositories from multiple load-balanced instances. It would simple create a new PVC for the new replica, and you would have different copies of the `/data`-mounted PV, each with different data. StatefulSet is a great mechanism for scaling RWO storage PVCs alongside replicas of an application that handles clustering natively, often requiring some amount of consistent identity between instances of pods (where "0" might be a leader, and any replica higher than "0" might be a follower, but "0" can be replaced and remain as "0"). In the case of Gitea replicas, the configuration you're documenting is attempting to offload most of the functionality to external HA components (DB, queue, cache, blob store, etc.) and keep the basic Git repositories (stored in `/data/git/gitea-repositories`) shared between multiple replicas. That will not work with a StatefulSet, even with RWX storage.
pat-s commented 2022-09-16 07:23:36 +00:00 (Migrated from gitea.com)

I think multiple stateful pods with mirrored RWO volumes compared to a central RWX is a choice and I see that some helm charts go this way.

Nothing prevents you or the chart from doing so but I think we should not tackle both at the same time. I also wonder if both should be supported at all within the chart, so far I have only seen charts which go with either one.

We're still early with HA here and my attempts are first of all to gather all important facts from various issues into a single document source.

In the case of Gitea replicas, the configuration you're documenting is attempting to offload most of the functionality to external HA components (DB, queue, cache, blob store, etc.) and keep the basic Git repositories (stored in /data/git/gitea-repositories) shared between multiple replicas.

Yes most must be outsourced to HA-ready components. The biggest issue is the code indexer as HA-ready ones often take many resources (e.g. elasticsearch).

That will not work with a StatefulSet, even with RWX storage.

Can you elaborate what you mean by "this will not work"? Do you mean it would only work for stateless replicas but stateful ones would require unique RWO copies?

I think multiple stateful pods with mirrored RWO volumes compared to a central RWX is a choice and I see that some helm charts go this way. Nothing prevents you or the chart from doing so but I think we should not tackle both at the same time. I also wonder if both should be supported at all within the chart, so far I have only seen charts which go with either one. We're still early with HA here and my attempts are first of all to gather all important facts from various issues into a single document source. > In the case of Gitea replicas, the configuration you're documenting is attempting to offload most of the functionality to external HA components (DB, queue, cache, blob store, etc.) and keep the basic Git repositories (stored in /data/git/gitea-repositories) shared between multiple replicas. Yes most must be outsourced to HA-ready components. The biggest issue is the code indexer as HA-ready ones often take many resources (e.g. `elasticsearch`). > That will not work with a StatefulSet, even with RWX storage. Can you elaborate what you mean by "this will not work"? Do you mean it would only work for stateless replicas but stateful ones would require unique RWO copies?
pat-s commented 2023-04-01 07:40:14 +00:00 (Migrated from gitea.com)

@solacelost I might have misunderstood your point earlier.

I agree that we should/must move from a StatefulSet to a stateless deployment to be able to use one central RWX PVC for repo storage.

This is a prerequisite we need to do in the chart before all the other points can be applied for a "real" HA experience.

@solacelost I might have misunderstood your point earlier. I agree that we should/must move from a StatefulSet to a stateless deployment to be able to use one central RWX PVC for repo storage. This is a prerequisite we need to do in the chart before all the other points can be applied for a "real" HA experience.
pat-s commented 2023-04-01 07:49:47 +00:00 (Migrated from gitea.com)

@justusbunsi I'd like to get it merged for better visibility and simplified updates, i.e. having a start in the first place.

I opted for a seperate .md to not make the main README too long while at the same time having a dedicated place for this distinct topic.

@justusbunsi I'd like to get it merged for better visibility and simplified updates, i.e. having a start in the first place. I opted for a seperate .md to not make the main README too long while at the same time having a dedicated place for this distinct topic.
justusbunsi (Migrated from gitea.com) reviewed 2023-04-01 07:50:02 +00:00
justusbunsi (Migrated from gitea.com) reviewed 2023-04-01 12:36:56 +00:00
docs/ha-setup.md
@ -46,1 +32,3 @@
An alternative to the two options above for the `ISSUE_INDEXER` is `"db"`, however we recommend to just go with `meilisearch` in this case and to not bother the DB with indexing.
- [elasticsearch](https://bitnami.com/stack/elasticsearch/helm) (4-8 GB)
- [opensearch](https://artifacthub.io/packages/helm/opensearch-project-helm-charts/opensearch) (4-8 GB)
- **not supported yet**: [manticore](https://manticoresearch.com/blog/manticore-alternative-to-elasticsearch/) (faster than `elasticsearch` & friends, lower resource usage)
justusbunsi (Migrated from gitea.com) commented 2023-04-01 12:36:56 +00:00

How should one read friends here? "Elasticsearch and similar apps" or "here me out, this uses lower resources"? The latter might be slang and could be removed, IMO.

How should one read `friends` here? "Elasticsearch and similar apps" or "here me out, this uses lower resources"? The latter might be slang and could be removed, IMO.
pat-s commented 2023-04-15 09:30:46 +00:00 (Migrated from gitea.com)

Now integrated in .

Now integrated in #437.

Pull request closed

This pull request cannot be reopened because the branch was deleted.
Sign in to join this conversation.
No Reviewers
justusbunsi
No Label
No Milestone
No project
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: lunny/helm-chart#350
No description provided.
Delete Branch "refs/pull/350/head"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?