lunny/helm-chart
1
0
Fork 0
Code Issues 36 Pull Requests 14 Actions Packages Projects Releases 93 Wiki Activity

Crazy number of Connection closed by logs #258

New Issue
Open
opened 2021-12-04 09:51:23 +00:00 by ghazan3 · 10 comments
ghazan3 commented 2021-12-04 09:51:23 +00:00 (Migrated from gitea.com)
Copy Link

Our log server record huge number of "Connection closed by 10.X.Y.Z port ABC" events. They are almost 3 events per second.

My first thought was bruteforce attempt but it is all from a local IP and happening every day all the time.

Is there a way to disable it? or figure out what is making these connections?

	Dec 4, 2021 @ 20:33:21.587	gitea	Connection closed by 10.104.0.7 port 36026
	Dec 4, 2021 @ 20:33:21.381	gitea	kex_exchange_identification: Connection closed by remote host
	Dec 4, 2021 @ 20:33:21.381	gitea	Connection closed by 10.104.0.12 port 35286
	Dec 4, 2021 @ 20:33:21.275	gitea	kex_exchange_identification: Connection closed by remote host
	Dec 4, 2021 @ 20:33:21.275	gitea	Connection closed by 10.104.0.3 port 63140
	Dec 4, 2021 @ 20:33:18.586	gitea	kex_exchange_identification: Connection closed by remote host
	Dec 4, 2021 @ 20:33:18.586	gitea	Connection closed by 10.104.0.7 port 36020
	Dec 4, 2021 @ 20:33:18.384	gitea	kex_exchange_identification: Connection closed by remote host
	Dec 4, 2021 @ 20:33:18.384	gitea	Connection closed by 10.104.0.12 port 35280
    Dec 4, 2021 @ 20:33:18.274	gitea	kex_exchange_identification: Connection closed by remote host
	Dec 4, 2021 @ 20:33:18.274	gitea	Connection closed by 10.104.0.3 port 63134
	Dec 4, 2021 @ 20:33:15.586	gitea	kex_exchange_identification: Connection closed by remote host
	Dec 4, 2021 @ 20:33:15.586	gitea	Connection closed by 10.104.0.7 port 36014
	Dec 4, 2021 @ 20:33:15.384	gitea	kex_exchange_identification: Connection closed by remote host
	Dec 4, 2021 @ 20:33:15.384	gitea	Connection closed by 10.104.0.12 port 35274
	Dec 4, 2021 @ 20:33:15.270	gitea	kex_exchange_identification: Connection closed by remote host
	Dec 4, 2021 @ 20:33:15.270	gitea	Connection closed by 10.104.0.3 port 63128
	Dec 4, 2021 @ 20:33:12.581	gitea	kex_exchange_identification: Connection closed by remote host
	Dec 4, 2021 @ 20:33:12.581	gitea	Connection closed by 10.104.0.7 port 36008

Relevant part of values.yaml

image:
  repository: gitea/gitea
  tag: 1.15.4
  
log:
      LEVEL: error
    log.name:
      COLORIZE: false
    metrics:
      enabled: false
      serviceMonitor:
        enabled: false

We had a dedicated loadbalancer on port 22 and map to ClusterIP of gitea-ssh service. We also use Cloudflare Spectrum service to tunnel SSH connections. Cloudflare does not show there are any connection to this service from public. So I am guessing these are internal connection or false logs?

Our log server record huge number of "Connection closed by 10.X.Y.Z port ABC" events. They are almost 3 events per second. My first thought was bruteforce attempt but it is all from a local IP and happening every day all the time. Is there a way to disable it? or figure out what is making these connections? ``` Dec 4, 2021 @ 20:33:21.587 gitea Connection closed by 10.104.0.7 port 36026 Dec 4, 2021 @ 20:33:21.381 gitea kex_exchange_identification: Connection closed by remote host Dec 4, 2021 @ 20:33:21.381 gitea Connection closed by 10.104.0.12 port 35286 Dec 4, 2021 @ 20:33:21.275 gitea kex_exchange_identification: Connection closed by remote host Dec 4, 2021 @ 20:33:21.275 gitea Connection closed by 10.104.0.3 port 63140 Dec 4, 2021 @ 20:33:18.586 gitea kex_exchange_identification: Connection closed by remote host Dec 4, 2021 @ 20:33:18.586 gitea Connection closed by 10.104.0.7 port 36020 Dec 4, 2021 @ 20:33:18.384 gitea kex_exchange_identification: Connection closed by remote host Dec 4, 2021 @ 20:33:18.384 gitea Connection closed by 10.104.0.12 port 35280 Dec 4, 2021 @ 20:33:18.274 gitea kex_exchange_identification: Connection closed by remote host Dec 4, 2021 @ 20:33:18.274 gitea Connection closed by 10.104.0.3 port 63134 Dec 4, 2021 @ 20:33:15.586 gitea kex_exchange_identification: Connection closed by remote host Dec 4, 2021 @ 20:33:15.586 gitea Connection closed by 10.104.0.7 port 36014 Dec 4, 2021 @ 20:33:15.384 gitea kex_exchange_identification: Connection closed by remote host Dec 4, 2021 @ 20:33:15.384 gitea Connection closed by 10.104.0.12 port 35274 Dec 4, 2021 @ 20:33:15.270 gitea kex_exchange_identification: Connection closed by remote host Dec 4, 2021 @ 20:33:15.270 gitea Connection closed by 10.104.0.3 port 63128 Dec 4, 2021 @ 20:33:12.581 gitea kex_exchange_identification: Connection closed by remote host Dec 4, 2021 @ 20:33:12.581 gitea Connection closed by 10.104.0.7 port 36008 ``` Relevant part of values.yaml ``` image: repository: gitea/gitea tag: 1.15.4 log: LEVEL: error log.name: COLORIZE: false metrics: enabled: false serviceMonitor: enabled: false ``` We had a dedicated loadbalancer on port 22 and map to ClusterIP of gitea-ssh service. We also use Cloudflare Spectrum service to tunnel SSH connections. Cloudflare does not show there are any connection to this service from public. So I am guessing these are internal connection or false logs?
👍 2
Sindvero commented 2022-01-07 21:14:47 +00:00 (Migrated from gitea.com)
Copy Link

I also have the same issue using the 1.15.8 release. It doesn't cause any issue that I can see nor connection, so I'm not sure where the logs are coming from.

I also have the same issue using the 1.15.8 release. It doesn't cause any issue that I can see nor connection, so I'm not sure where the logs are coming from.
pi3ch commented 2022-02-13 06:54:53 +00:00 (Migrated from gitea.com)
Copy Link

Any luck on this? Same issue with 1.16.1

Any luck on this? Same issue with 1.16.1
pi3ch commented 2022-02-18 00:54:56 +00:00 (Migrated from gitea.com)
Copy Link

I've disabled connection from public. Still getting the logs. It seems to be an inner cluster connection/prob against SSH service.

I've disabled connection from public. Still getting the logs. It seems to be an inner cluster connection/prob against SSH service.
pat-s commented 2022-03-05 16:04:43 +00:00 (Migrated from gitea.com)
Copy Link

Looks similar as #224

Looks similar as #224
👍 1
pi3ch commented 2022-03-27 13:39:51 +00:00 (Migrated from gitea.com)
Copy Link

These logs are related to openssh loglevel (not gitea). See a workaround at https://gitea.com/gitea/helm-chart/issues/224#issuecomment-694372

These logs are related to openssh loglevel (not gitea). See a workaround at https://gitea.com/gitea/helm-chart/issues/224#issuecomment-694372
pi3ch commented 2022-03-27 13:58:33 +00:00 (Migrated from gitea.com)
Copy Link

Connection attempts are from multiple nodes: cilium, csi or kube-proxy that are not SSH compatible.

Connection attempts are from multiple nodes: `cilium`, `csi` or `kube-proxy` that are not SSH compatible.
justusbunsi commented 2023-09-22 15:37:01 +00:00 (Migrated from gitea.com)
Copy Link

@pat-s Isn't this already resolved due to the configurable SSH log level or do I miss something obvious about this issue? #358

@pat-s Isn't this already resolved due to the configurable SSH log level or do I miss something obvious about this issue? #358
pi3ch commented 2023-09-22 23:26:05 +00:00 (Migrated from gitea.com)
Copy Link

@justusbunsi I can confirm by setting SSH log level (https://gitea.com/pi3ch/helm-chart/src/branch/main/values.yaml#L352) to FATAL the logs are not shown but I don't think this has addressed the underlying problem.

The problem only start to appear when there is LoadBalancer exposing SSH to public.

@justusbunsi I can confirm by setting SSH log level (https://gitea.com/pi3ch/helm-chart/src/branch/main/values.yaml#L352) to FATAL the logs are not shown but I don't think this has addressed the underlying problem. The problem only start to appear when there is LoadBalancer exposing SSH to public.
pat-s commented 2023-11-03 07:17:10 +00:00 (Migrated from gitea.com)
Copy Link

@justusbunsi I wonder if should change the default to ssh.logLevel: FATAL? Otherwise these messages only swamp the logs and might cause more concern than anything else.

@justusbunsi I wonder if should change the default to `ssh.logLevel: FATAL`? Otherwise these messages only swamp the logs and might cause more concern than anything else.
justusbunsi commented 2023-11-06 19:31:51 +00:00 (Migrated from gitea.com)
Copy Link

@justusbunsi I wonder if should change the default to ssh.logLevel: FATAL? Otherwise these messages only swamp the logs and might cause more concern than anything else.

Sounds good.
EDIT: I was too lax with my response. It would reduce the logged events but the underlying issue would still be there. And you don't see warnings anymore. Default log levels shouldn't mute warnings. IMO.

> @justusbunsi I wonder if should change the default to `ssh.logLevel: FATAL`? Otherwise these messages only swamp the logs and might cause more concern than anything else. ~~Sounds good.~~ EDIT: I was too lax with my response. It would reduce the logged events but the underlying issue would still be there. And you don't see warnings anymore. Default log levels shouldn't mute warnings. IMO.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
has/backport
in progress
invalid
kind/breaking
kind/bug
kind/build
kind/dependency
kind/deployment
kind/docs
kind/enhancement
kind/feature
kind/lint
kind/proposal
kind/question
kind/refactor
kind/security
kind/testing
kind/translation
kind/ui
need/backport
priority/critical
priority/low
priority/maybe
priority/medium
reviewed/duplicate
reviewed/invalid
reviewed/wontfix
skip-changelog
status/blocked
status/needs-feedback
status/needs-reviews
status/wip
upstream/gitea
upstream/other
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: lunny/helm-chart#258
No description provided.
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?