Skip to content

feat: use kaniko build and add container changes #14213

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
ohmayr merged 2 commits into from
Aug 11, 2025
Merged

feat: use kaniko build and add container changes #14213

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
e4f164b
feat: use kaniko build and add container changes
ohmayr Aug 11, 2025
fe23109
switch entrypoint to python3.9
ohmayr Aug 11, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
113 changes: 101 additions & 12 deletions .generator/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,10 +22,16 @@ RUN apt-get update && \
# Essential for compiling C code
build-essential \
# For downloading secure files
git \
wget \
ca-certificates \
# For running bazelisk commands
openjdk-17-jdk \
zip \
unzip \
# To avoid bazel error
# "python interpreter `python3` not found in PATH"
python3-dev \
# --- Critical libraries for a complete Python build ---
libssl-dev \
zlib1g-dev \
Expand Down Expand Up @@ -67,10 +73,39 @@ RUN wget https://github.com/bazelbuild/bazelisk/releases/download/${BAZELISK_VER
# Set the working directory for build-related tasks.
WORKDIR /app

# Create the group and user, but only if they don't already exist.
ARG UID=1000
ARG GID=1000

RUN if ! getent group $GID > /dev/null; then \
groupadd -g $GID myuser; \
fi && \
if ! getent passwd $UID > /dev/null; then \
useradd -u $UID -g $GID -ms /bin/bash myuser; \
fi

# Set ownership of the app directory now, before we copy files into it.
RUN mkdir -p /app && chown $UID:$GID /app

# We'll point both to the /bazel_cache directory which will be mounted as a volume.
ENV BAZELISK_HOME="/bazel_cache/bazelisk"
ENV BAZEL_HOME="/bazel_cache/bazel"

# Ensure the cache directories within the non-root user's context exist and are writable.
# This is crucial as Bazel creates subdirectories under BAZEL_HOME.
RUN mkdir -p ${BAZEL_HOME}/_bazel_ubuntu/cache/repos \
${BAZEL_HOME}/_bazel_ubuntu/output_base \
${BAZELISK_HOME} && \
chown -R $UID:$GID ${BAZEL_HOME} ${BAZELISK_HOME}

RUN /usr/local/bin/python3.9 -m venv bazel_env
RUN . bazel_env/bin/activate

RUN git clone https://github.com/googleapis/googleapis.git \
&& cd googleapis \
&& bazelisk --output_base=/bazel_cache/_bazel_ubuntu/output_base build --disk_cache=/bazel_cache/_bazel_ubuntu/cache/repos --incompatible_strict_action_env //google/cloud/language/v1:language-v1-py

# TODO(https://github.com/googleapis/librarian/issues/904): Install protoc for gencode.
# TODO(https://github.com/googleapis/librarian/issues/907): Install Python dependencies from requirements.in.
# TODO(https://github.com/googleapis/librarian/issues/905): Install Synthtool by cloning its repo.
# TODO(https://github.com/googleapis/librarian/issues/906): Clone googleapis and run bazelisk build.

# --- Final Stage ---
# This stage creates the lightweight final image, copying only the
Expand All @@ -82,21 +117,58 @@ FROM marketplace.gcr.io/google/ubuntu2404
RUN apt-get update && \
apt-get install -y --no-install-recommends \
ca-certificates \
git \
libssl3 \
zlib1g \
libbz2-1.0 \
libffi8 \
libsqlite3-0 \
libreadline8 \
# For running bazelisk commands
openjdk-17-jdk \
# To avoid bazel error
# "python interpreter `python3` not found in PATH"
python3-dev \
# To avoid bazel error
# "Cannot find gcc or CC; either correct your path or set the CC environment variable"
build-essential \
# To avoid bazel error
# unzip command not found
unzip \
&& apt-get clean && \
rm -rf /var/lib/apt/lists/*

# TODO(https://github.com/googleapis/librarian/issues/902): Create a dedicate non-root user and
# switch to the non-root user to run subsequent commands.
# Example:
# RUN groupadd --system --gid 1000 appgroup && \
# useradd --system --uid 1000 --gid appgroup appuser
# USER appuser
# Create the group and user, but only if they don't already exist.
# NOTE: A user needs to be configured by passing down the UID/GID with the
# docker build command to allow the user running the librarian CLI to have
# the permissions within the /app and /bazel_cache directories in the following way:
# DOCKER_BUILDKIT=1 docker build -f .generator/Dockerfile --build-arg UID=$(id -u) --build-arg GID=$(id -g) -t python-librarian-generator:latest .
ARG UID=1000
ARG GID=1000

RUN if ! getent group $GID > /dev/null; then \
groupadd -g $GID myuser; \
fi && \
if ! getent passwd $UID > /dev/null; then \
useradd -u $UID -g $GID -ms /bin/bash myuser; \
fi

# 2025/08/05 08:27:44 could not create directory /.cache/bazelisk: mkdir /.cache: permission denied
# Traceback (most recent call last):
# File "/app/./cli.py", line 147, in _build_bazel_target
# subprocess.run(
# File "/usr/local/lib/python3.9/subprocess.py", line 528, in run
# raise CalledProcessError(retcode, process.args,
# subprocess.CalledProcessError: Command '['bazelisk', '--output_base=/bazel_cache/_bazel_ubuntu/output_base', 'build', '--disk_cache=/bazel_cache/_bazel_ubuntu/cache/repos', '--incompatible_strict_action_env', '//google/cloud/language/v1beta2:language-v1beta2-py']' returned non-zero exit status 1
# We'll point both to the /bazel_cache directory which will be mounted as a volume.
ENV BAZELISK_HOME="/bazel_cache/bazelisk"
ENV BAZEL_HOME="/bazel_cache/bazel"
RUN mkdir -p ${BAZEL_HOME}/_bazel_ubuntu/cache/repos \
${BAZEL_HOME}/_bazel_ubuntu/output_base \
${BAZELISK_HOME} && \
chown -R $UID:$GID ${BAZEL_HOME} ${BAZELISK_HOME}

USER $UID

# Copy all Python interpreters, their pip executables, and their standard libraries from the builder.
COPY --from=builder /usr/local/bin/python3.9 /usr/local/bin/
Expand All @@ -114,11 +186,28 @@ COPY --from=builder /usr/local/lib/python3.12 /usr/local/lib/python3.12
COPY --from=builder /usr/local/bin/python3.13 /usr/local/bin/
COPY --from=builder /usr/local/lib/python3.13 /usr/local/lib/python3.13

# Copy the bazelisk executable from the builder.
COPY --from=builder /usr/local/bin/bazelisk /usr/local/bin/

# Copy bazel cache from the builder.
COPY --from=builder --chown=$UID:$GID /bazel_cache /bazel_cache

# Set the working directory in the container.
WORKDIR /app

# Create a virtual env and set the Path to fix the missing nox error
# when running the post processor changes.
RUN /usr/local/bin/python3.9 -m venv bazel_env
RUN . bazel_env/bin/activate

ENV PATH=/app/bazel_env/bin:$PATH

RUN git clone --depth 1 https://github.com/googleapis/synthtool.git /tmp/synthtool && \
bazel_env/bin/python3.9 -m pip install /tmp/synthtool nox && \
rm -rf /tmp/synthtool

# Copy the CLI script into the container.
COPY .generator/cli.py .
COPY --chown=$UID:$GID .generator/cli.py .
RUN chmod a+rx ./cli.py

# Set the entrypoint for the container to run the script.
ENTRYPOINT ["python3.11", "./cli.py"]
ENTRYPOINT ["python3.9", "./cli.py"]
36 changes: 15 additions & 21 deletions cloudbuild.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,28 +16,22 @@
# Reduce this timeout by moving the installation of Python runtimes to a separate base image
timeout: 7200s # 2 hours for the first uncached run, can be lowered later.
steps:
# STEP 1: Pull the latest version of the image to use as a cache.
# The '|| exit 0' part ensures the build doesn't fail if the image
# doesn't exist yet (e.g., on the very first run).
- name: 'gcr.io/cloud-builders/docker'
entrypoint: 'bash'
args: ['-c', 'docker pull gcr.io/$PROJECT_ID/python-librarian-generator:latest || exit 0']

# STEP 2: Build the new image, using the pulled image as a cache source.
- name: 'gcr.io/cloud-builders/docker'
# A single step using the Kaniko executor to build and cache
- name: 'gcr.io/kaniko-project/executor:latest'
args:
- 'build'
- '--tag=gcr.io/$PROJECT_ID/python-librarian-generator:latest'
- '--cache-from=gcr.io/$PROJECT_ID/python-librarian-generator:latest'
- '--file=.generator/Dockerfile'
- '.'
# Specifies the Dockerfile path
- '--dockerfile=.generator/Dockerfile'
# Specifies the build context directory
- '--context=.'
# The final destination for the image
- '--destination=gcr.io/$PROJECT_ID/python-librarian-generator:latest'
# Enables Kaniko's remote registry caching
- '--cache=true'
# (Optional but recommended) Sets a time-to-live for cache layers
- '--cache-ttl=24h'

# STEP 3: Push the newly built image to the registry so it can be used
# as the cache for the next run. Cloud Build does this automatically if
# the image is listed in the 'images' section.
images:
- 'gcr.io/$PROJECT_ID/python-librarian-generator:latest'
# The 'images' section is no longer needed because Kaniko pushes the image itself.

# This section automatically creates a storage bucket for storing docker build logs.
options:
default_logs_bucket_behavior: REGIONAL_USER_OWNED_BUCKET
default_logs_bucket_behavior: REGIONAL_USER_OWNED_BUCKET
machineType: E2_HIGHCPU_32
Loading