lunny/helm-chart
1
0
Fork 0
Code Issues 36 Pull Requests 14 Actions Packages Projects Releases 93 Wiki Activity

Making rootless containers the default breaks SSH keys on the server side #487

New Issue
Open
opened 2023-08-21 12:38:22 +00:00 by rkevin · 4 comments
rkevin commented 2023-08-21 12:38:22 +00:00 (Migrated from gitea.com)
Copy Link

Hi! I recently upgraded gitea via the helm chart. After an upgrade from helm chart 8.3.0 to 9.1.0, the SSH key for gitea changed and I got a WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED warning. ssh-keyscan tells me the server only understands ssh-rsa and not ssh-ed25519. An ls -la in /data/ssh shows

total 48
drwxrws---  2 root   rkevin 4096 Aug 21 04:16 .
drwxrwsr-x 16 rkevin rkevin 4096 Aug 21 04:12 ..
-rw-rw----  1 root   rkevin 3243 Aug 21 04:16 gitea.rsa
-rw-rw----  1 root   rkevin  725 Aug 21 04:16 gitea.rsa.pub
-rw-rw----  1 root   rkevin 1381 Feb  6  2022 ssh_host_dsa_key
-rw-rw----  1 root   rkevin  602 Feb  6  2022 ssh_host_dsa_key.pub
-rw-rw----  1 root   rkevin  505 Feb  6  2022 ssh_host_ecdsa_key
-rw-rw----  1 root   rkevin  174 Feb  6  2022 ssh_host_ecdsa_key.pub
-rw-rw----  1 root   rkevin  399 Feb  6  2022 ssh_host_ed25519_key
-rw-rw----  1 root   rkevin   94 Feb  6  2022 ssh_host_ed25519_key.pub
-rw-rw----  1 root   rkevin 1823 Feb  6  2022 ssh_host_rsa_key
-rw-rw----  1 root   rkevin  394 Feb  6  2022 ssh_host_rsa_key.pub

which seems like gitea is no longer using the standard ssh_host_* files and is using gitea.rsa instead.

I found that START_SSH_SERVER has been set to true in app.ini, and it was repeatedly set to true after every pod restart (which was confusing since the documentation assured me it was false by default). After some digging, I found this to be the culprit, which sets START_SSH_SERVER if we use rootless containers, which seems to be true by default since 9.0.0. Sure enough, after I set image.rootless: false, the issue went away and I didn't have to modify .ssh/known_hosts on all my devices.

I found a discussion here, which seems to suggest this isn't an issue, but it was for me. I imagine the lack of user reports might be because people just rotated their .ssh/known_hosts to get rid of the warning. If there's some way for the internal SSH server to read the same ssh_host_* files, that'd be great. Otherwise, I think there should be a warning under the upgrade section on this.

Hi! I recently upgraded gitea via the helm chart. After an upgrade from helm chart 8.3.0 to 9.1.0, the SSH key for gitea changed and I got a `WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED` warning. `ssh-keyscan` tells me the server only understands `ssh-rsa` and not `ssh-ed25519`. An `ls -la` in `/data/ssh` shows ``` total 48 drwxrws--- 2 root rkevin 4096 Aug 21 04:16 . drwxrwsr-x 16 rkevin rkevin 4096 Aug 21 04:12 .. -rw-rw---- 1 root rkevin 3243 Aug 21 04:16 gitea.rsa -rw-rw---- 1 root rkevin 725 Aug 21 04:16 gitea.rsa.pub -rw-rw---- 1 root rkevin 1381 Feb 6 2022 ssh_host_dsa_key -rw-rw---- 1 root rkevin 602 Feb 6 2022 ssh_host_dsa_key.pub -rw-rw---- 1 root rkevin 505 Feb 6 2022 ssh_host_ecdsa_key -rw-rw---- 1 root rkevin 174 Feb 6 2022 ssh_host_ecdsa_key.pub -rw-rw---- 1 root rkevin 399 Feb 6 2022 ssh_host_ed25519_key -rw-rw---- 1 root rkevin 94 Feb 6 2022 ssh_host_ed25519_key.pub -rw-rw---- 1 root rkevin 1823 Feb 6 2022 ssh_host_rsa_key -rw-rw---- 1 root rkevin 394 Feb 6 2022 ssh_host_rsa_key.pub ``` which seems like gitea is no longer using the standard `ssh_host_*` files and is using `gitea.rsa` instead. I found that `START_SSH_SERVER` has been set to `true` in `app.ini`, and it was repeatedly set to true after every pod restart (which was confusing since [the documentation](https://docs.gitea.com/administration/config-cheat-sheet?_highlight=start_ssh_server#server-server) assured me it was `false` by default). After some digging, I found [this](https://gitea.com/gitea/helm-chart/src/commit/5e148748ce241ef11c9fe5649bdeac27fc7c8121/templates/_helpers.tpl#L338-L342) to be the culprit, which sets `START_SSH_SERVER` if we use rootless containers, which seems to be `true` by default since `9.0.0`. Sure enough, after I set `image.rootless: false`, the issue went away and I didn't have to modify `.ssh/known_hosts` on all my devices. I found a discussion [here](https://gitea.com/gitea/helm-chart/issues/432), which seems to suggest this isn't an issue, but it was for me. I imagine the lack of user reports might be because people just rotated their `.ssh/known_hosts` to get rid of the warning. If there's some way for the internal SSH server to read the same `ssh_host_*` files, that'd be great. Otherwise, I think there should be a warning under the upgrade section on this.
👍 1
pat-s commented 2023-08-21 14:22:32 +00:00 (Migrated from gitea.com)
Copy Link

Hi Kevin

Yes, the chart switched to the rootless image lately. And this image uses a different, built-in SSH server.

I found a discussion here, which seems to suggest this isn't an issue, but it was for me.

Sorry to hear and thanks for reporting!

I imagine the lack of user reports might be because people just rotated their .ssh/known_hosts to get rid of the warning.

Could be even though I would have expected more or earlier reports of this.

I found that START_SSH_SERVER has been set to true in app.ini, and it was repeatedly set to true after every pod restart (which was confusing since the documentation assured me it was false by default). After some digging, I found this to be the culprit, which sets START_SSH_SERVER if we use rootless containers,

Yes that's a bit of a tricky thing where the helm-chart differs from the actual default for good reason.

If there's some way for the internal SSH server to read the same ssh_host_* files, that'd be great.

I am not sure the built-in one can handle the new ed25519 format which is what your error message indicates.

If there's some way for the internal SSH server to read the same ssh_host_* files, that'd be great. Otherwise, I think there should be a warning under the upgrade section on this.

That would definitely make sense overall, maybe something that should be discussed on the project level. In the meantime I've added your report here to the upgrade notes, thanks again!

Hi Kevin Yes, the chart switched to the rootless image lately. And this image uses a different, built-in SSH server. > I found a discussion here, which seems to suggest this isn't an issue, but it was for me. Sorry to hear and thanks for reporting! > I imagine the lack of user reports might be because people just rotated their .ssh/known_hosts to get rid of the warning. Could be even though I would have expected more or earlier reports of this. > I found that START_SSH_SERVER has been set to true in app.ini, and it was repeatedly set to true after every pod restart (which was confusing since the documentation assured me it was false by default). After some digging, I found this to be the culprit, which sets START_SSH_SERVER if we use rootless containers, Yes that's a bit of a tricky thing where the helm-chart differs from the actual default for good reason. > If there's some way for the internal SSH server to read the same ssh_host_* files, that'd be great. I am not sure the built-in one can handle the new `ed25519` format which is what your error message indicates. > If there's some way for the internal SSH server to read the same ssh_host_* files, that'd be great. Otherwise, I think there should be a warning under the upgrade section on this. That would definitely make sense overall, maybe something that should be discussed on the project level. In the meantime I've added your report here to the upgrade notes, thanks again!
rkevin commented 2023-08-21 22:51:31 +00:00 (Migrated from gitea.com)
Copy Link

Cool, thanks for adding the note in the README! I'll leave this open in case you want to use it to track an upstream fix, or you can close it since the breaking change is now reflected in the upgrade notes.

Cool, thanks for adding the note in the README! I'll leave this open in case you want to use it to track an upstream fix, or you can close it since the breaking change is now reflected in the upgrade notes.
👍 2
danmanners commented 2023-09-16 22:03:47 +00:00 (Migrated from gitea.com)
Copy Link

I'm running into this issue as well. My values files at this time for the deployment can be found here.

When I'm running more than a single node, the fingerprint from ssh-keyscan switches between as many replicas exist.

hwatch ssh-keyscan git.homelab.danmanners.com

Tried switching over to the privileged images/deployment, but it didn't seem to work for me. Happy to help troubleshoot however I can!

I'm running into this issue as well. My values files at this time for the deployment [can be found here](https://github.com/danmanners/homelab-kube-cluster/blob/4fd4c8e/manifests/workloads/gitea/values.yaml). When I'm running more than a single node, the fingerprint from `ssh-keyscan` switches between as many replicas exist. > `hwatch ssh-keyscan git.homelab.danmanners.com` Tried switching over to the privileged images/deployment, but it didn't seem to work for me. Happy to help troubleshoot however I can!
pat-s commented 2023-09-18 08:43:51 +00:00 (Migrated from gitea.com)
Copy Link

@danmanners Please open a new issue (with logs and a thorough description) as your issue seems to be unrelated to rootless/rootful images.

@danmanners Please open a new issue (with logs and a thorough description) as your issue seems to be unrelated to rootless/rootful images.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
has/backport
in progress
invalid
kind/breaking
kind/bug
kind/build
kind/dependency
kind/deployment
kind/docs
kind/enhancement
kind/feature
kind/lint
kind/proposal
kind/question
kind/refactor
kind/security
kind/testing
kind/translation
kind/ui
need/backport
priority/critical
priority/low
priority/maybe
priority/medium
reviewed/duplicate
reviewed/invalid
reviewed/wontfix
skip-changelog
status/blocked
status/needs-feedback
status/needs-reviews
status/wip
upstream/gitea
upstream/other
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: lunny/helm-chart#487
No description provided.
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?