Skip to content

node:22-alpine builds fail with "alpine-baselayout: failed to commit var/run" when using Kaniko on Kubernetes #2431

@onurridvanoglu

Description

@onurridvanoglu

Environment

  • Platform: linux/amd64 (GKE Kubernetes nodes, running Kaniko builds)
  • Docker Version: Kaniko executor (v1.x, running as a Kubernetes pod in GitLab CI)
  • Node.js Version: 22.x / 20.x
  • Image Tag: node:22-alpine (Alpine 3.23), node:20-alpine (Alpine 3.23)

Expected Behavior

Running apk update && apk upgrade in a Dockerfile based on node:22-alpine should complete successfully, as it did with previous Alpine versions (3.20, 3.21, 3.22). This is a common pattern in Dockerfiles to ensure the base image has the latest security patches before installing additional packages.

  FROM node:22-alpine
  RUN apk update && apk upgrade && \
      apk add --no-cache python3 make g++ gcc libc6-compat    

Current Behavior

When building the above Dockerfile with Kaniko inside a Kubernetes pod, apk upgrade fails during the alpine-baselayout package upgrade (3.7.1-r8 → 3.7.2-r0):

 (1/3) Upgrading alpine-baselayout-data (3.7.1-r8 -> 3.7.2-r0)
 (2/3) Upgrading alpine-baselayout (3.7.1-r8 -> 3.7.2-r0)                                                                                                 
   Executing alpine-baselayout-3.7.2-r0.pre-upgrade                                                                                                       
   * rm: can't remove 'var/run/secrets/kubernetes.io/serviceaccount/token': Read-only file system                                                         
   * rm: can't remove 'var/run/secrets/kubernetes.io/serviceaccount/namespace': Read-only file system                                                     
   * rm: can't remove 'var/run/secrets/kubernetes.io/serviceaccount/ca.crt': Read-only file system
   * rm: can't remove 'var/run/secrets/kubernetes.io/serviceaccount/..data': Read-only file system                                                        
   * rm: can't remove 'var/run/secrets/kubernetes.io/serviceaccount/..2026_03_27_08_27_57.1948645492/ca.crt': Read-only file system
   * rm: can't remove 'var/run/secrets/kubernetes.io/serviceaccount/..2026_03_27_08_27_57.1948645492/token': Read-only file system                        
   * rm: can't remove 'var/run/secrets/kubernetes.io/serviceaccount/..2026_03_27_08_27_57.1948645492/namespace': Read-only file system                    
 ERROR: alpine-baselayout-3.7.2-r0: failed to commit var/run: Is a directory                                                                              
 1 error; 23.2 MiB in 31 packages                                                                                                                         
 error building image: error building stage: failed to execute command: waiting for process to exit: exit status
1

The alpine-baselayout 3.7.2 upgrade script attempts to restructure /var/run (converting it from a directory to a symlink to /run). In a Kubernetes environment, /var/run/secrets/kubernetes.io/serviceaccount/ is a read-only projected volume mount. Kaniko runs unprivileged and without a full container runtime, so the build process sees the host pod's filesystem mounts. The upgrade script cannot remove or modify these read-only mounted files, causing the entire build to fail.

Possible Solution

Since the Node.js Alpine images are based on the upstream Alpine base image, there are a few possible approaches:

  1. Pin the Alpine version in the image tags: The node:22-alpine tag currently resolves to Alpine 3.23, which includes the problematic alpine-baselayout upgrade path. Keeping Alpine 3.22 as the default (or providing clearer documentation about this breaking change) would prevent unexpected build failures.
  2. Document the Kaniko/Kubernetes incompatibility: At minimum, the known issue with apk upgrade in Kaniko-based builds on Kubernetes should be documented, so users are aware that node:XX-alpine images based on Alpine 3.23 may break in these environments.
  3. Upstream fix in Alpine: This is ultimately an Alpine packaging issue where alpine-baselayout 3.7.2's pre-upgrade script doesn't gracefully handle read-only filesystem mounts. However, since the Node.js image inherits this behavior, users encounter it here first.

Steps to Reproduce

  1. Set up a Kubernetes cluster with a GitLab CI runner (or any CI system that uses Kaniko for in-cluster Docker builds)
  2. Create a Dockerfile:
FROM node:22-alpine                                                                                                                                      
RUN apk update && apk upgrade && \
    apk add --no-cache python3 make g++ gcc libc6-compat
  1. Build the image using Kaniko inside a Kubernetes pod:
    /kaniko/executor --context . --dockerfile Dockerfile --destination <registry>/test:latest --cache=true
  2. The build fails with the alpine-baselayout error shown above.

Workaround: Pin to an older Alpine version:
FROM node:22-alpine3.22
or remove apk upgrade from the Dockerfile:
RUN apk add --no-cache python3 make g++ gcc libc6-compat

Additional Information

  • This issue affects any node:XX-alpine image that resolves to Alpine 3.23 when built with Kaniko on Kubernetes. Both node:20-alpine and node:22-alpine are affected.
  • Pinning to node:22-alpine3.22 or node:20-alpine3.20 resolves the issue immediately.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions