Unable to run docker containers in dind on bare metal #4138
Description
I installed linuxkit on bare metal, merging several example yaml files (docker, sshd, cadvisor, node_exporter, ...). I got it to work once I built and insmod'ed a few kernel modules.
config:
kernel:
image: linuxkit/kernel:6.6.71
cmdline: "console=tty0"
ucode: intel-ucode.cpio
init:
- linuxkit/init:8eea386739975a43af558eec757a7dcb3a3d2e7b
- linuxkit/runc:667e7ea2c426a2460ca21e3da065a57dbb3369c9
- linuxkit/containerd:a988a1a8bcbacc2c0390ca0c08f949e2b4b5915d
- linuxkit/ca-certificates:7b32a26ca9c275d3ef32b11fe2a83dbd2aee2fdb
- linuxkit/firmware:bfc7802f909c4b760de5dd2bc02a7f52e86b78f7
- linuxkit/firmware-all:c8b72e1e1c1aa512836ae2374998121fa120f592
onboot:
- name: sysctl
image: linuxkit/sysctl:5f56434b81004b50b47ed629b222619168c2bcdf
# adds realtek, r8169, usb-storage, ums-realtek kernel modules for ethernet and sd card storage
- name: custom_kernel_modules
image: linuxkit/my/custom-modules
- name: dhcpcd
image: linuxkit/dhcpcd:157df9ef45a035f1542ec2270e374f18efef98a5
command: ["/sbin/dhcpcd", "--nobackground", "-f", "/dhcpcd.conf", "-1"]
- name: sysfs
image: linuxkit/sysfs:7345172dbf4d436c861adfc27150af474194289b
# ext4 partition on sd card storage, added to the raw-efi image with losetup and sfdisk after building it with linuxkit
- name: mount
image: linuxkit/mount:cb8caa72248f7082fc2074ce843d53cdc15df04a
command: ["/usr/bin/mountie", "/dev/sda2", "/var/lib/docker"]
services:
- name: getty
image: linuxkit/getty:05eca453695984a69617f1f1f0bcdae7f7032967
env:
- INSECURE=true
- name: rngd
image: linuxkit/rngd:1a18f2149e42a0a1cb9e7d37608a494342c26032
- name: ntpd
image: linuxkit/openntpd:f99c4117763480815553b72022b426639a13ce86
- name: sshd
image: linuxkit/sshd:9bdd85427ef99640276d97a32a7a3cc31bb017b3
binds.add:
- /root/.ssh:/root/.ssh
- name: docker
image: docker:20.10.6-dind
#image: docker:28.3.0-dind
capabilities:
- all
net: host
mounts:
- type: cgroup
options: ["rw", "nosuid", "noexec", "nodev", "relatime"]
binds:
- /etc/resolv.conf:/etc/resolv.conf
- /var/lib/docker:/var/lib/docker
- /var/run:/var/run
- /lib/modules:/lib/modules
- /etc/docker/daemon.json:/etc/docker/daemon.json
command: ["/usr/local/bin/docker-init", "/usr/local/bin/dockerd"]
- name: node_exporter
image: linuxkit/node_exporter:1415b52c08ddc5799b2fc83cf3f080c56c3ff5a9
- name: cadvisor
image: linuxkit/cadvisor:8dfefe0f9593ba21aca5d08fadac16de907d470d
files:
- path: root/.ssh/authorized_keys
contents: |
ssh-ed25519 [...] [...]
mode: "0600"
optional: true
- path: var/lib/docker
directory: true
- path: etc/docker/daemon.json
contents: '{"debug": true, "hosts": ["tcp://0.0.0.0:2375", "unix:///var/run/docker.sock"]}'
mode: "0644"
- path: etc/linuxkit-config
metadata: yaml
- path: /etc/containerd/runtime-config.toml
contents: |
cliopts="--log-level trace"
stderr="/var/log/containerd.err.log"
stdout="/var/log/containerd.out.log"
mode: "0644"I then tried to run a docker container inside the docker service. Tested both by ctr task exec'ing into the service then running docker directly, and by remotely connecting through the tcp interface. Tested multiple images, such as alpine:latest or nginx:latest.
In both cases, the container seems to get created but refuses to start properly. I see errors in the logs such as stream copy error: reading from a closed fifo and /proc/7/exe: no such file or directory for docker, and failed to read init pid file [...] open /run/containerd/io.containerd.runtime.v2.task/moby/[...]/init.pid: no such file or directory for containerd.
docker ps -a then shows the container with status Created, but it's not running.
Tested both with docker:20.10.6-dind and docker:28.3.0-dind with the same result (small difference, with the newer image it is /bin/sh that is not found first (though it exists in the image) instead of /proc/7/exe).
/var/log/docker.log:
tmp_docker.log
/var/log/containerd.err.log:
tmp_containerd.log
Any idea on what could be causing this?