about act_runner run in rootless docker #170

Closed
opened 2023-05-02 10:18:10 +00:00 by UnknownO · 8 comments
UnknownO commented 2023-05-02 10:18:10 +00:00 (Migrated from gitea.com)

hello, about act_runner, I have some questions
For the security of the system, I use rootless docker, act_runner is run at user runner
I can start act_runner at ssh screen like /usr/local/bin/act_runner daemon
but when I write a service file to made act_runner startup with system. The problem arises:
time="2023-05-02T09:28:45Z" level=error msg="failed to obtain container engine info: Cannot connect to the Docker daemon at unix:///run/user/1001/docker.sock. Is the docker daemon running

This is my service file

[Unit]
Description=Gitea Action Runner
After=syslog.target
After=network.target

[Service]
Type=simple
User=runner
Group=runner
WorkingDirectory=/home/runner
ExecStart=/usr/local/bin/act_runner daemon
Environment=DOCKER_HOST=unix:///run/user/1001/docker.sock

[Install]
WantedBy=multi-user.target

Is my service file wrong? or act_runner not support rootless docker?

PS: Sorry for my bad English

hello, about act_runner, I have some questions For the security of the system, I use rootless docker, act_runner is run at user `runner` I can start act_runner at ssh screen like `/usr/local/bin/act_runner daemon` but when I write a service file to made act_runner startup with system. The problem arises: ` time="2023-05-02T09:28:45Z" level=error msg="failed to obtain container engine info: Cannot connect to the Docker daemon at unix:///run/user/1001/docker.sock. Is the docker daemon running` This is my service file ``` [Unit] Description=Gitea Action Runner After=syslog.target After=network.target [Service] Type=simple User=runner Group=runner WorkingDirectory=/home/runner ExecStart=/usr/local/bin/act_runner daemon Environment=DOCKER_HOST=unix:///run/user/1001/docker.sock [Install] WantedBy=multi-user.target ``` Is my service file wrong? or act_runner not support rootless docker? PS: Sorry for my bad English
wolfogre commented 2023-05-04 03:34:05 +00:00 (Migrated from gitea.com)

I can start act_runner at ssh screen like /usr/local/bin/act_runner daemon

Was the user you were using at the time also the "runner" user?

Sorry I'm not familiar with rootless docker, but it looks like the runner user can't access /run/user/1001/docker.sock

> I can start act_runner at ssh screen like `/usr/local/bin/act_runner daemon` Was the user you were using at the time also the "runner" user? Sorry I'm not familiar with rootless docker, but it looks like the runner user can't access `/run/user/1001/docker.sock`
UnknownO commented 2023-05-04 03:56:27 +00:00 (Migrated from gitea.com)

I can start act_runner at ssh screen like /usr/local/bin/act_runner daemon

Was the user you were using at the time also the "runner" user?

Sorry I'm not familiar with rootless docker, but it looks like the runner user can't access /run/user/1001/docker.sock

hello, when i direct run the act_runner with user runner it's run good. but when i use service to startup, it's told this error, in service file, i assigned user runner to run it in service config
is my service config wrong?

> > I can start act_runner at ssh screen like `/usr/local/bin/act_runner daemon` > > Was the user you were using at the time also the "runner" user? > > Sorry I'm not familiar with rootless docker, but it looks like the runner user can't access `/run/user/1001/docker.sock` hello, when i direct run the act_runner with user runner it's run good. but when i use service to startup, it's told this error, in service file, i assigned user runner to run it in service config is my service config wrong?
UnknownO commented 2023-05-04 03:59:39 +00:00 (Migrated from gitea.com)

i alreadly set run user in service file
User=runner
Group=runner

and i'm sure that runner user has permissions to access /run/user/1001/docker.sock

i alreadly set run user in service file User=runner Group=runner and i'm sure that runner user has permissions to access /run/user/1001/docker.sock
ccureau commented 2023-05-20 14:35:37 +00:00 (Migrated from gitea.com)

I have a preliminary rootless container ready for general testing. I've only done basic checks on it (i.e., it builds my repositories on my server) without issue both in local Docker and Kubernetes. So far it seems a bit slow, but I'm not sure if that's a factor from the machine I'm using or rootless Docker in general.

Sadly, 'rootless' docker still requires elevated privileges to the Docker daemon.

Here's my working branch: https://gitea.com/ccureau/act_runner/src/branch/feat/rootless-runner

Do a make docker, and use the act_runner_rootless:nightly image.

An example runner using Docker:

docker run -d --privileged -e DOCKER_HOST=unix:///run/user/1000/docker.sock -e GITEA_INSTANCE_URL=<your instance> -e GITEA_RUNNER_REGISTRATION_TOKEN=<your token> gitea/act_runner_rootless:nightly
I have a preliminary rootless container ready for general testing. I've only done basic checks on it (i.e., it builds my repositories on my server) without issue both in local Docker and Kubernetes. So far it seems a bit slow, but I'm not sure if that's a factor from the machine I'm using or rootless Docker in general. Sadly, 'rootless' docker still requires elevated privileges to the Docker daemon. Here's my working branch: https://gitea.com/ccureau/act_runner/src/branch/feat/rootless-runner Do a `make docker`, and use the `act_runner_rootless:nightly` image. An example runner using Docker: ``` docker run -d --privileged -e DOCKER_HOST=unix:///run/user/1000/docker.sock -e GITEA_INSTANCE_URL=<your instance> -e GITEA_RUNNER_REGISTRATION_TOKEN=<your token> gitea/act_runner_rootless:nightly ```
ccureau commented 2023-05-21 13:01:29 +00:00 (Migrated from gitea.com)

@UnknownO @wolfogre

I was able to create a completely rootless Debian 11 VM today which survives reboots. 🎉 The trick seems to be to not use the system-level systemd, but instead keep everything on the rootless user's level.

Brief overview of what I did:

  • Install docker-ce and docker-ce-rootless-extras as normal, but disable the global systemd services
  • Create a new user rootlessand log in as this user. This avoids the could not connect to bus issue.
  • Run dockerd-rootless-setuptool.sh install
  • Add these env variables to /home/rootless/.bashrc
    • XDG_RUNTIME_DIR=/home/rootless/.docker/run
    • PATH=/home/rootless/bin:$PATH
    • DOCKER_HOST=unix:///run/user/1001/docker.sock
  • Enable docker unit as the rootless user (systemctl --user enable docker)
  • Reboot
  • Create a unit for act_runner as /home/rootless/.config/systemd/user/act_runner.service. I just copied the existing docker.service to act_runner.service and modified.
  • Create a symlink for the act_runner.service under /home/rootless/.config/systemd/default.target.wants
  • Add Environment=DOCKER_HOST=unix:///run/user/1001/docker.sock to the unit
  • Add WorkingDirectory=/home/runner/data or wherever yo put it.
  • Create a data directory (I just kept everything under /home/rootless)
  • Register the runner in the data directory
  • systemctl --user daemon-reload
  • systemctl --user enable --now act_runner.service

Regardless, I'll write all this up for inclusion in the README and make a PR.

@UnknownO @wolfogre I was able to create a completely rootless Debian 11 VM today which survives reboots. 🎉 The trick seems to be to not use the system-level systemd, but instead keep everything on the rootless user's level. Brief overview of what I did: - Install `docker-ce` and `docker-ce-rootless-extras` as normal, but disable the global systemd services - Create a new user `rootless`and log in as this user. This avoids the `could not connect to bus` issue. - Run `dockerd-rootless-setuptool.sh install` - Add these env variables to `/home/rootless/.bashrc` - `XDG_RUNTIME_DIR=/home/rootless/.docker/run` - `PATH=/home/rootless/bin:$PATH` - `DOCKER_HOST=unix:///run/user/1001/docker.sock` - Enable docker unit as the rootless user (`systemctl --user enable docker`) - Reboot - Create a unit for act_runner as `/home/rootless/.config/systemd/user/act_runner.service`. I just copied the existing docker.service to act_runner.service and modified. - Create a symlink for the `act_runner.service` under `/home/rootless/.config/systemd/default.target.wants` - Add `Environment=DOCKER_HOST=unix:///run/user/1001/docker.sock` to the unit - Add `WorkingDirectory=/home/runner/data` or wherever yo put it. - Create a data directory (I just kept everything under /home/rootless) - Register the runner in the data directory - `systemctl --user daemon-reload` - `systemctl --user enable --now act_runner.service` Regardless, I'll write all this up for inclusion in the README and make a PR.
cakekindel commented 2023-06-27 18:50:03 +00:00 (Migrated from gitea.com)

@ccureau could you share your act_runner config.yml? I've been getting really frustrated trying to get the default job image to connect to the host's rootless docker socket.

config.yml

log:
  level: 'info'

runner:
  file: '.runner'
  capacity: 1
  timeout: '3h'
  insecure: false
  fetch_timeout: '5s'
  fetch_interval: '2s'

container:
  valid_volumes: ['/run/user/1001/docker.sock']
  docker_host: 'unix:///run/user/1001/docker.sock'

host: stat /var/run/docker.sock; stat /run/user/1001/docker.sock

> stat /var/run/docker.sock
  File: /var/run/docker.sock
  Size: 0         	Blocks: 0          IO Block: 4096   socket
Device: 17h/23d	Inode: 736         Links: 1
Access: (0660/srw-rw----)  Uid: (    0/    root)   Gid: (  998/  docker)
Access: 2023-06-24 01:17:14.326079292 +0000
Modify: 2023-06-22 20:53:16.360000000 +0000
Change: 2023-06-22 20:53:16.360000000 +0000
 Birth: -
> stat /run/user/1001/docker.sock
  File: /run/user/1001/docker.sock
  Size: 0         	Blocks: 0          IO Block: 4096   socket
Device: 27h/39d	Inode: 29          Links: 1
Access: (0777/srwxrwxrwx)  Uid: ( 1001/gitea_runner)   Gid: (232069/ UNKNOWN)
Access: 2023-06-27 18:33:06.872341390 +0000
Modify: 2023-06-27 18:32:35.572224764 +0000
Change: 2023-06-27 18:32:38.864237035 +0000
 Birth: -

workflow: stat /var/run/docker.sock

> stat /var/run/docker.sock
  File: /var/run/docker.sock
  Size: 0         	Blocks: 0          IO Block: 4096   socket
Device: 17h/23d	Inode: 736         Links: 1
Access: (0660/srw-rw----)  Uid: (65534/  nobody)   Gid: (65534/ nogroup)
Access: 2023-06-24 01:17:14.326079292 +0000
Modify: 2023-06-22 20:53:16.360000000 +0000
Change: 2023-06-22 20:53:16.360000000 +0000
 Birth: -

/home/gitea_runner/.config/systemd/user/act_runner.service

[Unit]
Description=Gitea Actions runner
Documentation=https://gitea.com/gitea/data
After=docker.service

[Service]
Environment=PATH=/home/gitea_runner/bin:/sbin:/usr/sbin:/home/gitea_runner/bin:/home/gitea_runner/bin:/home/gitea_runner/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
Environment=DOCKER_HOST=unix:///run/user/1001/docker.sock
ExecStart=/usr/bin/act_runner daemon -c /home/gitea_runner/config.yml
ExecReload=/bin/kill -s HUP $MAINPID
WorkingDirectory=/home/gitea_runner/data
TimeoutSec=0
RestartSec=2
Restart=always
StartLimitBurst=3
StartLimitInterval=60s
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
TasksMax=infinity
Delegate=yes
Type=notify
NotifyAccess=all
KillMode=mixed

[Install]
WantedBy=default.target

have also tried:

container:
  privileged: true
  options: '-e DOCKER_HOST=unix:///run/user/1001/docker.sock -v /run/user/1001/docker.sock:/run/user/1001/docker.sock'
  valid_volumes: ['/run/user/1001/docker.sock']
  docker_host: 'unix:///run/user/1001/docker.sock'

which doesn't actually mount the socket to /run/user/1001/docker.sock on the container yielding not found errors

minimal repro of failing workflow:
https://git.orionkindel.com/orion/act-runner-dind-demo

@ccureau could you share your act_runner config.yml? I've been getting really frustrated trying to get the default job image to connect to the host's rootless docker socket. <details> <summary> `config.yml` </summary> ```yaml log: level: 'info' runner: file: '.runner' capacity: 1 timeout: '3h' insecure: false fetch_timeout: '5s' fetch_interval: '2s' container: valid_volumes: ['/run/user/1001/docker.sock'] docker_host: 'unix:///run/user/1001/docker.sock' ``` </details> <details> <summary> host: `stat /var/run/docker.sock; stat /run/user/1001/docker.sock` </summary> ```sh > stat /var/run/docker.sock File: /var/run/docker.sock Size: 0 Blocks: 0 IO Block: 4096 socket Device: 17h/23d Inode: 736 Links: 1 Access: (0660/srw-rw----) Uid: ( 0/ root) Gid: ( 998/ docker) Access: 2023-06-24 01:17:14.326079292 +0000 Modify: 2023-06-22 20:53:16.360000000 +0000 Change: 2023-06-22 20:53:16.360000000 +0000 Birth: - > stat /run/user/1001/docker.sock File: /run/user/1001/docker.sock Size: 0 Blocks: 0 IO Block: 4096 socket Device: 27h/39d Inode: 29 Links: 1 Access: (0777/srwxrwxrwx) Uid: ( 1001/gitea_runner) Gid: (232069/ UNKNOWN) Access: 2023-06-27 18:33:06.872341390 +0000 Modify: 2023-06-27 18:32:35.572224764 +0000 Change: 2023-06-27 18:32:38.864237035 +0000 Birth: - ``` </details> <details> <summary> workflow: `stat /var/run/docker.sock` </summary> ```sh > stat /var/run/docker.sock File: /var/run/docker.sock Size: 0 Blocks: 0 IO Block: 4096 socket Device: 17h/23d Inode: 736 Links: 1 Access: (0660/srw-rw----) Uid: (65534/ nobody) Gid: (65534/ nogroup) Access: 2023-06-24 01:17:14.326079292 +0000 Modify: 2023-06-22 20:53:16.360000000 +0000 Change: 2023-06-22 20:53:16.360000000 +0000 Birth: - ``` </details> <details> <summary> `/home/gitea_runner/.config/systemd/user/act_runner.service` </summary> ``` [Unit] Description=Gitea Actions runner Documentation=https://gitea.com/gitea/data After=docker.service [Service] Environment=PATH=/home/gitea_runner/bin:/sbin:/usr/sbin:/home/gitea_runner/bin:/home/gitea_runner/bin:/home/gitea_runner/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games Environment=DOCKER_HOST=unix:///run/user/1001/docker.sock ExecStart=/usr/bin/act_runner daemon -c /home/gitea_runner/config.yml ExecReload=/bin/kill -s HUP $MAINPID WorkingDirectory=/home/gitea_runner/data TimeoutSec=0 RestartSec=2 Restart=always StartLimitBurst=3 StartLimitInterval=60s LimitNOFILE=infinity LimitNPROC=infinity LimitCORE=infinity TasksMax=infinity Delegate=yes Type=notify NotifyAccess=all KillMode=mixed [Install] WantedBy=default.target ``` </details> have also tried: ```yaml container: privileged: true options: '-e DOCKER_HOST=unix:///run/user/1001/docker.sock -v /run/user/1001/docker.sock:/run/user/1001/docker.sock' valid_volumes: ['/run/user/1001/docker.sock'] docker_host: 'unix:///run/user/1001/docker.sock' ``` which doesn't actually mount the socket to `/run/user/1001/docker.sock` on the container yielding not found errors minimal repro of failing workflow: https://git.orionkindel.com/orion/act-runner-dind-demo
cakekindel commented 2023-06-27 23:36:51 +00:00 (Migrated from gitea.com)

my god, i thought that container.valid_volumes was in 0.2.0 😭

switching to a nightly binary fixed my issue, i now have unprivileged runner containers that can use the act_runner user's docker daemon 🎉

my god, i thought that `container.valid_volumes` was in `0.2.0` 😭 switching to a nightly binary fixed my issue, i now have unprivileged runner containers that can use the act_runner user's docker daemon :tada:
hoangln94masan commented 2024-10-08 06:41:56 +00:00 (Migrated from gitea.com)

-- i have move all my repos to fogejo, it have more details docs and more security auto-test --
Follow rootless concept of Podman. I think its more secure than mapping /var/run/docker.sock because only a rootless user (which have no root priviledges of this runner machine is effected, because Podman not run in daemon mode as root like docker.
Here is my step to make act_runner instance, with container platform is podman on rootless, no priviledges linux user:

-Step 1: install Podman - follow this: https://github.com/containers/podman/blob/main/docs/tutorials/rootless_tutorial.md

-Step 2: create a rootless user, must have /bin/bash and home dir, should named it as instance, orga or a repo its belong to. Add this to .bashrc or .profile in $HOME of this user

export XDG_CONFIG_HOME=$HOME/.config
export XDG_DATA_HOME=$HOME/.local/share
export XDG_RUNTIME_DIR=/run/user/$UID
export DOCKER_HOST=DOCKER_HOST=unix:///run/user/$UID/podman/podman.sock

-Step 3: sudo or sudo -i to root to enable linger (systemd service able to start and run when user not login)
loginctl enable-linger <user-name>

-Step 4: enable podman rootless socket of this user
systemctl --user start podman.socket && systemctl --user enable podman.socket

-Step 5: Download runner binary. I have created a .gitea dir on $HOME, which have config.yml file and data dir which have act_runner binary and .runner file
mkdir -p $HOME/.gitea/data

-Step 6: Create config.yml file on $HOME/.gitea with content as: replace {{ user_uid }} with your user $(id -u) user id number

log:
  level: 'info'

runner:
  file: '.runner'
  capacity: 1
  envs:
    DOCKER_HOST: 'unix:///run/user/{{ user_uid }}/podman/podman.sock'
  timeout: '3h'
  insecure: false
  fetch_timeout: '5s'
  fetch_interval: '2s'

labels:
  - "{{ gitea_runner_label }}:docker://node:16-bullseye:{{ gitea_runner_label }}"

container:
  privileged: true
  options: '-e DOCKER_HOST=unix:///run/user/{{ user_uid }}/podman/podman.sock -v /run/user/{{ user_uid }}/podman/podman.sock:/run/user/{{ user_uid }}/podman/podman.sock'
  workdir_parent: "{{ act_user_home }}/.gitea"
  valid_volumes: ['/run/user/{{ user_uid }}/podman/podman.sock']
  docker_host: 'unix:///run/user/{{ user_uid }}/podman/podman.sock'
  force_pull: false

hosts:
  workdir_parent: "{{ act_user_home }}/.cache/act" 

-Step 7: Register runner to generate .runner file on $HOME/.gitea/data
./act_runner --config ../config.yml register

-Step 8: create user systemd service file on $HOME/.config/systemd/user, mine is act_runner.service, chmod is 644

[Unit]
Description=Gitea Actions Runner Service
Documentation=https://gitea.com/gitea/act_runner
After=podman.socket

[Service]
WorkingDirectory={{ act_user_home }}/.gitea/data
ExecStart={{ act_user_home }}/.gitea/data/act_runner daemon -c {{ act_user_home }}/.gitea/config.yml
ExecReload=/bin/kill -s HUP $MAINPID
TimeoutSec=0
RestartSec=10
Restart=on-failure
Environment=DOCKER_HOST="unix:///run/user/{{ user_uid }}/podman/podman.sock" 

[Install]
WantedBy=default.target

-Step 9: Start and enable service, voila !!!!
systemctl --user daemon-reload systemctl --user start act_runner.service systemctl --user enable act_runner.service

Use on CICD with:


jobs:
  build:
    runs-on: <runner-label>

    container:
      image: docker.io/node:16
      options: --privileged
-- i have move all my repos to fogejo, it have more details docs and more security auto-test -- Follow rootless concept of Podman. I think its more secure than mapping /var/run/docker.sock because only a rootless user (which have no root priviledges of this runner machine is effected, because Podman not run in daemon mode as root like docker. Here is my step to make act_runner instance, with container platform is podman on rootless, no priviledges linux user: -Step 1: install Podman - follow this: https://github.com/containers/podman/blob/main/docs/tutorials/rootless_tutorial.md -Step 2: create a rootless user, must have /bin/bash and home dir, should named it as instance, orga or a repo its belong to. Add this to .bashrc or .profile in $HOME of this user ``` export XDG_CONFIG_HOME=$HOME/.config export XDG_DATA_HOME=$HOME/.local/share export XDG_RUNTIME_DIR=/run/user/$UID export DOCKER_HOST=DOCKER_HOST=unix:///run/user/$UID/podman/podman.sock ``` -Step 3: sudo or sudo -i to root to enable linger (systemd service able to start and run when user not login) `loginctl enable-linger <user-name>` -Step 4: enable podman rootless socket of this user `systemctl --user start podman.socket && systemctl --user enable podman.socket` -Step 5: Download runner binary. I have created a .gitea dir on $HOME, which have config.yml file and data dir which have act_runner binary and .runner file `mkdir -p $HOME/.gitea/data` -Step 6: Create config.yml file on $HOME/.gitea with content as: replace {{ user_uid }} with your user $(id -u) user id number ``` log: level: 'info' runner: file: '.runner' capacity: 1 envs: DOCKER_HOST: 'unix:///run/user/{{ user_uid }}/podman/podman.sock' timeout: '3h' insecure: false fetch_timeout: '5s' fetch_interval: '2s' labels: - "{{ gitea_runner_label }}:docker://node:16-bullseye:{{ gitea_runner_label }}" container: privileged: true options: '-e DOCKER_HOST=unix:///run/user/{{ user_uid }}/podman/podman.sock -v /run/user/{{ user_uid }}/podman/podman.sock:/run/user/{{ user_uid }}/podman/podman.sock' workdir_parent: "{{ act_user_home }}/.gitea" valid_volumes: ['/run/user/{{ user_uid }}/podman/podman.sock'] docker_host: 'unix:///run/user/{{ user_uid }}/podman/podman.sock' force_pull: false hosts: workdir_parent: "{{ act_user_home }}/.cache/act" ``` -Step 7: Register runner to generate .runner file on $HOME/.gitea/data `./act_runner --config ../config.yml register` -Step 8: create user systemd service file on $HOME/.config/systemd/user, mine is act_runner.service, chmod is 644 ``` [Unit] Description=Gitea Actions Runner Service Documentation=https://gitea.com/gitea/act_runner After=podman.socket [Service] WorkingDirectory={{ act_user_home }}/.gitea/data ExecStart={{ act_user_home }}/.gitea/data/act_runner daemon -c {{ act_user_home }}/.gitea/config.yml ExecReload=/bin/kill -s HUP $MAINPID TimeoutSec=0 RestartSec=10 Restart=on-failure Environment=DOCKER_HOST="unix:///run/user/{{ user_uid }}/podman/podman.sock" [Install] WantedBy=default.target ``` -Step 9: Start and enable service, voila !!!! `systemctl --user daemon-reload systemctl --user start act_runner.service systemctl --user enable act_runner.service` Use on CICD with: ``` jobs: build: runs-on: <runner-label> container: image: docker.io/node:16 options: --privileged ```
Sign in to join this conversation.