Probes do not catch gitea 500 errors #118

New Issue

2021-02-22T10:31:13Z

viceice commented

2021-02-22 10:31:13 +00:00

(Migrated from gitea.com)

Currently the static liveness and readiness probes only do a tcp check. So they don't fail if gitea returns 500 errors.

This can happen if the external redis service gets a new cluster ip (because of upgrade). Gitea is not automatically resolving the new ip and logs no route to host errors.

2021/02/22 11:30:17 ...ue/queue_bytefifo.go:127:readToChan() [E] redis: task Error on Pop: dial tcp 10.42.3.24:6379: connect: no route to host
2021/02/22 11:30:17 ...ue/queue_bytefifo.go:127:readToChan() [E] redis: push_update Error on Pop: dial tcp 10.42.3.24:6379: connect: no route to host
2021/02/22 11:30:17 ...ue/queue_bytefifo.go:127:readToChan() [E] unique-redis: repo_stats_update Error on Pop: dial tcp 10.42.3.25:6379: connect: no route to host
2021/02/22 11:30:17 ...ue/queue_bytefifo.go:127:readToChan() [E] redis: notification-service Error on Pop: dial tcp 10.42.1.66:6379: connect: no route to host
2021/02/22 11:30:17 ...ue/queue_bytefifo.go:127:readToChan() [E] redis: code_indexer Error on Pop: dial tcp 10.42.3.25:6379: connect: no route to host
2021/02/22 11:30:18 ...ue/queue_bytefifo.go:127:readToChan() [E] redis: notification-service Error on Pop: dial tcp 10.42.1.66:6379: connect: no route to host
2021/02/22 11:30:18 ...ue/queue_bytefifo.go:127:readToChan() [E] redis: task Error on Pop: dial tcp 10.42.3.24:6379: connect: no route to host
2021/02/22 11:30:18 ...ue/queue_bytefifo.go:127:readToChan() [E] unique-redis: pr_patch_checker Error on Pop: dial tcp 10.42.3.24:6379: connect: no route to host
2021/02/22 11:30:18 ...ue/queue_bytefifo.go:127:readToChan() [E] redis: push_update Error on Pop: dial tcp 10.42.3.24:6379: connect: no route to host
2021/02/22 11:30:18 ...ue/queue_bytefifo.go:127:readToChan() [E] redis: issue_indexer Error on Pop: dial tcp 10.42.3.25:6379: connect: no route to host
2021/02/22 11:30:18 ...ue/queue_bytefifo.go:127:readToChan() [E] redis: code_indexer Error on Pop: dial tcp 10.42.3.25:6379: connect: no route to host

I've create a pr, so users can override the probes as required.

Currently the static liveness and readiness probes only do a tcp check. So they don't fail if gitea returns 500 errors. This can happen if the external redis service gets a new cluster ip (because of upgrade). Gitea is not automatically resolving the new ip and logs `no route to host` errors. ``` 2021/02/22 11:30:17 ...ue/queue_bytefifo.go:127:readToChan() [E] redis: task Error on Pop: dial tcp 10.42.3.24:6379: connect: no route to host 2021/02/22 11:30:17 ...ue/queue_bytefifo.go:127:readToChan() [E] redis: push_update Error on Pop: dial tcp 10.42.3.24:6379: connect: no route to host 2021/02/22 11:30:17 ...ue/queue_bytefifo.go:127:readToChan() [E] unique-redis: repo_stats_update Error on Pop: dial tcp 10.42.3.25:6379: connect: no route to host 2021/02/22 11:30:17 ...ue/queue_bytefifo.go:127:readToChan() [E] redis: notification-service Error on Pop: dial tcp 10.42.1.66:6379: connect: no route to host 2021/02/22 11:30:17 ...ue/queue_bytefifo.go:127:readToChan() [E] redis: code_indexer Error on Pop: dial tcp 10.42.3.25:6379: connect: no route to host 2021/02/22 11:30:18 ...ue/queue_bytefifo.go:127:readToChan() [E] redis: notification-service Error on Pop: dial tcp 10.42.1.66:6379: connect: no route to host 2021/02/22 11:30:18 ...ue/queue_bytefifo.go:127:readToChan() [E] redis: task Error on Pop: dial tcp 10.42.3.24:6379: connect: no route to host 2021/02/22 11:30:18 ...ue/queue_bytefifo.go:127:readToChan() [E] unique-redis: pr_patch_checker Error on Pop: dial tcp 10.42.3.24:6379: connect: no route to host 2021/02/22 11:30:18 ...ue/queue_bytefifo.go:127:readToChan() [E] redis: push_update Error on Pop: dial tcp 10.42.3.24:6379: connect: no route to host 2021/02/22 11:30:18 ...ue/queue_bytefifo.go:127:readToChan() [E] redis: issue_indexer Error on Pop: dial tcp 10.42.3.25:6379: connect: no route to host 2021/02/22 11:30:18 ...ue/queue_bytefifo.go:127:readToChan() [E] redis: code_indexer Error on Pop: dial tcp 10.42.3.25:6379: connect: no route to host ``` I've create a pr, so users can override the probes as required.

👍 1

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: lunny/helm-chart#118