Skip to content

Commit 29222a1

Browse files
cgwalterscgwalters
committed
Add build+test infra mirroring bootc
This introduces a Justfile and Dockerfile to enable building rpm-ostree from source in a container, following the pattern established in the bootc project. See the `Justfile` for key entrypoints. Those are now used in a new GHA flow, which we'll try to move things over to. A key difference though vs bootc is because rpm-ostree has a lot of C++ too we use sccache which greatly speeds things up across incremental rebuilds. Just one side cleanup of this is before it was *terribly* painful and manual to hack on `test-container.sh`, and now it's easy, fast and optimized. Assisted-by: Claude Code (Sonnet 4.5) Signed-off-by: Colin Walters <walters@verbum.org>
1 parent 88218c8 commit 29222a1

File tree

12 files changed

+393
-15
lines changed

12 files changed

+393
-15
lines changed

‎.copr/Makefile‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,7 @@ srpm:
55
# if we have a git repo with remotes, fetch tags so `git describe` gives a nice NEVRA when
66
# building the RPM
77
if git remote | grep origin; then git fetch origin --tags; fi
8-
git submodule update --init --recursive
8+
if [ -d .git ]; then git submodule update --init --recursive; fi
99
# Our primary CI build goes via RPM rather than direct to binaries
1010
# to better test that path, including our vendored spec file, etc.
1111
make -C packaging -f Makefile.dist-packaging srpm

‎.dockerignore‎

Lines changed: 54 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,54 @@
1-
.cosa
2-
target
3-
compose-cache/
1+
# Exclude everything by default, then include just what we need
2+
# Especially note this means that .git is not included, and not tests/
3+
# to avoid spurious rebuilds.
4+
*
5+
6+
# Autotools build files
7+
!Makefile*.am
8+
!Makefile-*.am
9+
!Makefile*.inc
10+
!Makefile.bindings
11+
!configure.ac
12+
!autogen.sh
13+
14+
# Generated C++/Rust bridge files (checked into git)
15+
!rpmostree-cxxrs.h
16+
!rpmostree-cxxrs.cxx
17+
!rpmostree-cxxrsutil.hpp
18+
19+
# Build configuration
20+
!buildutil/
21+
!build-aux/
22+
!m4/
23+
24+
# Source code
25+
!src/
26+
!rust/
27+
28+
# Rust build files
29+
!Cargo.toml
30+
!Cargo.lock
31+
!build.rs
32+
!.cargo/
33+
34+
# Git submodules (needed by autogen.sh)
35+
!libglnx/
36+
!libdnf/
37+
38+
# Build system integration
39+
!packaging/
40+
!ci/
41+
!.copr/
42+
43+
# Test data for integration tests
44+
!tests/
45+
46+
# Documentation (for man pages, etc.)
47+
!docs/
48+
!man/
49+
50+
# Shell completion
51+
!completion/
52+
53+
# API documentation generation
54+
!api-doc/

‎.github/workflows/container-build.yml‎

Lines changed: 58 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,58 @@
1+
# Container Build CI Workflow
2+
#
3+
# Builds rpm-ostree from source in a container using the Dockerfile and Justfile.
4+
# This workflow follows the pattern established in bootc for containerized builds.
5+
name: Container Build
6+
7+
permissions:
8+
actions: read
9+
10+
on:
11+
push:
12+
branches: [main]
13+
pull_request:
14+
branches: [main]
15+
workflow_dispatch: {}
16+
17+
env:
18+
CARGO_TERM_COLOR: always
19+
20+
concurrency:
21+
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
22+
cancel-in-progress: true
23+
24+
jobs:
25+
# Build container and run basic validation
26+
build-and-validate:
27+
runs-on: ubuntu-24.04
28+
strategy:
29+
fail-fast: false
30+
matrix:
31+
base_image:
32+
- name: fedora-42
33+
image: quay.io/fedora/fedora-bootc:42
34+
# TODO: Enable CentOS Stream 10 once tests support it
35+
# - name: centos-10
36+
# image: quay.io/centos-bootc/centos-bootc:stream10
37+
38+
steps:
39+
- name: Checkout repository
40+
uses: actions/checkout@v5
41+
with:
42+
submodules: true
43+
44+
- name: Bootc Ubuntu Setup
45+
uses: ./.github/actions/bootc-ubuntu-setup
46+
47+
- name: Run validation
48+
run: |
49+
just validate
50+
51+
- name: Build container
52+
run: |
53+
set -xeuo pipefail
54+
just build --build-arg=base=${{ matrix.base_image.image }}
55+
56+
- name: Run container integration tests
57+
run: |
58+
just test-container-integration

‎Cargo.toml‎

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -134,3 +134,28 @@ bin-unit-tests = []
134134
sanitizers = []
135135

136136
default = []
137+
138+
[lints]
139+
workspace = true
140+
141+
[workspace.lints.rust]
142+
# Absolutely must handle errors
143+
unused_must_use = "forbid"
144+
missing_debug_implementations = "deny"
145+
# Feel free to comment this one out locally during development of a patch.
146+
dead_code = "deny"
147+
148+
# We aren't using these yet
149+
# [workspace.lints.rust]
150+
# unsafe_code = "deny"
151+
# missing_docs = "deny"
152+
153+
[workspace.lints.clippy]
154+
disallowed_methods = "deny"
155+
# These should only be in local code
156+
dbg_macro = "deny"
157+
todo = "deny"
158+
# These two are in my experience the lints which are most likely
159+
# to trigger, and among the least valuable to fix.
160+
needless_borrow = "allow"
161+
needless_borrows_for_generic_args = "allow"

‎Dockerfile‎

Lines changed: 123 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,123 @@
1+
# Build this project from source and write the updated content
2+
# (i.e. /usr/bin/rpm-ostree and related binaries) to a new derived container
3+
# image. See the `Justfile` for an example
4+
#
5+
# Use e.g. --build-arg=base=quay.io/centos-bootc/centos-bootc:stream10 to target
6+
# CentOS instead.
7+
8+
ARG base=quay.io/fedora/fedora-bootc:42
9+
10+
# This first image captures a snapshot of the source code,
11+
# note all the exclusions in .dockerignore.
12+
FROM scratch as src
13+
COPY . /src
14+
15+
# This is basically a no-op now, but we could make any other final tweaks we want
16+
# here.
17+
FROM $base as base
18+
19+
# Fetch sccache
20+
FROM base as sccache
21+
ARG SCCACHE_VERSION=0.8.2
22+
# Install sccache for compiler caching
23+
RUN <<EORUN
24+
set -xeuo pipefail
25+
target=$(arch)-unknown-linux-musl
26+
v=sccache-v${SCCACHE_VERSION}-${target}
27+
curl -fSsL "https://github.com/mozilla/sccache/releases/download/v${SCCACHE_VERSION}/${v}.tar.gz" \
28+
| tar xz -C /usr/local/bin --strip-components=1 "${v}/sccache"
29+
chmod +x /usr/local/bin/sccache
30+
EORUN
31+
32+
# This image installs build deps, pulls in our source code, and installs updated
33+
# rpm-ostree binaries in /out. The intention is that the target rootfs is extracted from /out
34+
# back into a final stage (without the build deps etc) below.
35+
FROM base as build
36+
# This installs our package dependencies, and we want to cache it independently of the rest.
37+
# Basically we don't want changing a .rs file to blow out the cache of packages. So we only
38+
# copy files necessary for dependency installation.
39+
COPY packaging /tmp/packaging
40+
COPY ci/installdeps.sh ci/libbuild.sh /tmp/ci/
41+
RUN <<EORUN
42+
set -xeuo pipefail
43+
. /usr/lib/os-release
44+
case $ID in
45+
centos|rhel) dnf config-manager --set-enabled crb;;
46+
fedora) dnf -y install dnf-utils 'dnf5-command(builddep)';;
47+
esac
48+
# Handle version skew, upgrade core dependencies
49+
dnf -y distro-sync ostree{,-libs} libmodulemd
50+
# Install build requirements
51+
cd /tmp && ./ci/installdeps.sh
52+
rm /tmp/{packaging,ci} -rf
53+
EORUN
54+
COPY --from=sccache /usr/local/bin/* /usr/local/bin/
55+
# Now copy the rest of the source
56+
COPY --from=src /src /src
57+
WORKDIR /src
58+
# See https://www.reddit.com/r/rust/comments/126xeyx/exploring_the_problem_of_faster_cargo_docker/
59+
# We aren't using the full recommendations there, just the simple bits.
60+
# First step, ensure we have the crates downloaded
61+
RUN --mount=type=cache,target=/src/target --mount=type=cache,target=/var/roothome cargo fetch
62+
# Then this all runs without networking
63+
RUN --mount=type=cache,target=/src/target --mount=type=cache,target=/var/roothome --mount=type=cache,target=/var/cache/sccache --network=none <<EORUN
64+
set -xeuo pipefail
65+
# Configure sccache for C/C++ and Rust compilation caching
66+
export SCCACHE_DIR=/var/cache/sccache
67+
export CC="sccache gcc"
68+
export CXX="sccache g++"
69+
export RUSTC_WRAPPER=sccache
70+
sccache --show-stats || true
71+
env NOCONFIGURE=1 ./autogen.sh
72+
./configure --prefix=/usr --libdir=/usr/lib64 --sysconfdir=/etc
73+
make -j $(nproc)
74+
make install DESTDIR=/out
75+
sccache --show-stats
76+
EORUN
77+
78+
# This just does syntax checking and basic validation
79+
FROM build as validate
80+
RUN grep -vEe '^#' ci/packages-build-extra.txt | xargs dnf -y install
81+
RUN --mount=type=cache,target=/src/target --mount=type=cache,target=/var/roothome --mount=type=cache,target=/var/cache/sccache --network=none <<EORUN
82+
set -xeuo pipefail
83+
# Use sccache in validate stage too
84+
export SCCACHE_DIR=/var/cache/sccache
85+
export CC="sccache gcc"
86+
export CXX="sccache g++"
87+
export RUSTC_WRAPPER=sccache
88+
# Only gate on correctness and a few specific lints by default
89+
cargo clippy -- -A clippy::all -D clippy::correctness -D clippy::suspicious -D clippy::disallowed-methods -Dunused_imports -Ddead_code
90+
# Basic syntax checks
91+
make check-local
92+
EORUN
93+
94+
# The final image that derives from the original base and adds the release binaries
95+
FROM base as final
96+
# Create a layer that is our new binaries
97+
COPY --from=build /out/ /
98+
# Only in this containerfile, inject a file which signifies
99+
# this comes from this development image. This can be used in
100+
# tests to know we're doing upstream CI.
101+
RUN touch /usr/lib/.rpm-ostree-dev-stamp
102+
103+
# Integration test build
104+
FROM build as integration-build
105+
RUN <<EORUN
106+
set -xeuo pipefail
107+
grep -vEe '^#' ci/packages-build-extra.txt | xargs dnf -y install
108+
grep -vEe '^#' ci/integration-runtime.txt | xargs dnf -y install
109+
EORUN
110+
# Copy test scripts
111+
COPY ci/test-container.sh /out/usr/bin/rpm-ostree-test-container.sh
112+
# Copy test data if it exists
113+
COPY --from=src /src/tests /usr/share/rpm-ostree/tests
114+
RUN <<EORUN
115+
set -xeuo pipefail
116+
make -C tests/kolainst install DESTDIR=/out
117+
EORUN
118+
119+
FROM final as integration
120+
COPY ci/ /run/ci/
121+
RUN grep -vEe '^#' /run/ci/integration-runtime.txt | xargs dnf -y install
122+
COPY --from=integration-build /out/ /
123+
FROM final

‎Justfile‎

Lines changed: 46 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,46 @@
1+
# The default entrypoint to working on this project.
2+
# Commands here typically wrap e.g. `podman build` or
3+
# other tools which might launch local virtual machines.
4+
#
5+
# See also `Makefile`. Commands which end in `-local`
6+
# skip containerization (and typically just proxy `make`).
7+
#
8+
# Rules written here are *often* used by the Github Action flows,
9+
# and should support being configurable where that makes sense (e.g.
10+
# the `build` rule supports being provided a base image).
11+
12+
# --------------------------------------------------------------------
13+
14+
# Build the container image from current sources.
15+
# Note commonly you might want to override the base image via e.g.
16+
# `just build --build-arg=base=quay.io/fedora/fedora-bootc:42`
17+
build *ARGS:
18+
podman build --jobs=4 -t localhost/rpm-ostree {{ARGS}} .
19+
20+
# Perform validation (build, linting) in a container build environment
21+
validate:
22+
podman build --jobs=4 --target validate .
23+
24+
# Directly run validation using host tools
25+
validate-local:
26+
make check
27+
28+
# Build the integration test container image
29+
build-integration:
30+
podman build --jobs=4 --target integration -t localhost/rpm-ostree-integration .
31+
32+
# Run container integration tests
33+
test-container-integration: build-integration
34+
podman run --rm localhost/rpm-ostree-integration rpm-ostree-test-container.sh
35+
36+
# Build and run a shell in the build environment for debugging
37+
shell:
38+
podman build --jobs=4 --target build -t localhost/rpm-ostree-build .
39+
podman run --rm -it localhost/rpm-ostree-build /bin/bash
40+
41+
# Build RPMs in a container
42+
rpm *ARGS:
43+
podman build --jobs=4 --ignorefile packaging/.dockerignore -f packaging/Containerfile {{ARGS}} -t localhost/rpm-ostree-rpm .
44+
podman create --name rpm-ostree-rpm-tmp localhost/rpm-ostree-rpm
45+
podman cp rpm-ostree-rpm-tmp:/ .
46+
podman rm rpm-ostree-rpm-tmp

‎ci/integration-runtime.txt‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
# Packages needed for integration tests
2+
rsync
3+
rpm-build
4+
createrepo_c

‎ci/packages-build-extra.txt‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
# Extra packages needed in the build container for Fedora derivatives
2+
# that aren't actually used to build the code necessarily, but
3+
# are used by targets in the Dockerfile.
4+
rustfmt
5+
clippy
6+
git-core
7+
jq
8+
rsync

‎ci/test-container.sh‎

Lines changed: 0 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -9,17 +9,6 @@ fatal() {
99

1010
versionid=$(. /usr/lib/os-release && echo $VERSION_ID)
1111

12-
# This allows running this test in a podman container locally by running
13-
# `SELF_BOOTSTRAP=1 ci/test-container.sh`.
14-
if [ -n "${SELF_BOOTSTRAP:-}" ]; then
15-
rm -rf "$PWD/installtree"
16-
make install DESTDIR="$PWD/installtree"
17-
make -C tests/kolainst install DESTDIR="$PWD/installtree"
18-
exec podman run -ti --rm --security-opt=label=disable -v "$PWD":/var/srv -w /var/srv \
19-
quay.io/fedora/fedora-coreos:stable sh -c \
20-
'rsync -rlv installtree/ / && /var/srv/ci/test-container.sh'
21-
fi
22-
2312
# Test overrides
2413
# These hardcoded versions can be kept until Fedora GC's them
2514
ignition_url_suffix=2.17.0/4.fc40/x86_64/ignition-2.17.0-4.fc40."$(arch)".rpm

‎packaging/.dockerignore‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
# For RPM builds, we need minimal exclusions
2+
# since we need .git for creating source tarballs
3+
target/
4+
.cosa/
5+
compose-cache/

0 commit comments

Comments
 (0)