Skip to content

ROX-33792: build operator binary outside Docker for faster builds#19655

Merged
janisz merged 4 commits intomasterfrom
ROX-33792-undockerize-operator-build
Mar 31, 2026
Merged

ROX-33792: build operator binary outside Docker for faster builds#19655
janisz merged 4 commits intomasterfrom
ROX-33792-undockerize-operator-build

Conversation

@janisz
Copy link
Copy Markdown
Contributor

@janisz janisz commented Mar 27, 2026

Description

Build operator binary once in pre-build-go-binaries job alongside other Go binaries, then reuse it when building Docker images. This eliminates duplicate compilation and leverages GitHub Actions' Go module cache and Go builder for multiple binaries.

User-facing documentation

Testing and quality

  • the change is production ready: the change is GA, or otherwise the functionality is gated by a feature flag
  • CI results are inspected

Automated testing

  • added unit tests
  • added e2e tests
  • added regression tests
  • added compatibility tests
  • modified existing tests

How I validated my change

CI

@janisz janisz requested review from a team as code owners March 27, 2026 13:37
@janisz janisz requested review from mclasmeier and removed request for a team March 27, 2026 13:37
Copy link
Copy Markdown
Contributor

@sourcery-ai sourcery-ai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey - I've found 1 issue, and left some high level feedback:

  • The docker-build target’s CI detection relies on CI being exactly "true"; consider relaxing this to a non-empty check (e.g., [ -n "$$CI" ]) or documenting the expectation, to avoid surprises in other CI environments.
  • The new prebuilt flow assumes bin/linux_${GOARCH}/stackrox-operator is present in the Docker build context and not filtered by .dockerignore/.containerignore; it would be good to double‑check or explicitly comment in those files that this binary must remain included for prebuilt.Dockerfile to work.
Prompt for AI Agents
Please address the comments from this code review:

## Overall Comments
- The `docker-build` target’s CI detection relies on `CI` being exactly `"true"`; consider relaxing this to a non-empty check (e.g., `[ -n "$$CI" ]`) or documenting the expectation, to avoid surprises in other CI environments.
- The new prebuilt flow assumes `bin/linux_${GOARCH}/stackrox-operator` is present in the Docker build context and not filtered by `.dockerignore`/`.containerignore`; it would be good to double‑check or explicitly comment in those files that this binary must remain included for `prebuilt.Dockerfile` to work.

## Individual Comments

### Comment 1
<location path="operator/Makefile" line_range="393-394" />
<code_context>
+		-f $< \
+		..
+
+.PHONY: docker-build-prebuilt
+docker-build-prebuilt: prebuilt.Dockerfile ## Build docker image with pre-built operator binary.
 	BUILDKIT_PROGRESS=plain ../scripts/docker-build.sh \
 		-t ${IMG} \
</code_context>
<issue_to_address>
**🚨 issue (security):** Align docker-build-prebuilt prerequisites with docker-build-full (e.g., smuggled-status-sh) to preserve the same checks.

After the refactor, `docker-build`’s former prerequisite chain (`build/Dockerfile.gen smuggled-status-sh`) only exists on `docker-build-full`, while `docker-build-prebuilt` depends solely on `prebuilt.Dockerfile`. In CI the prebuilt path will run most often, so any checks or side effects in `smuggled-status-sh` (e.g., license/provenance validation) will be skipped. Please add `smuggled-status-sh` (and any other needed prerequisites) to `docker-build-prebuilt`, or extract them into a shared prerequisite so both build modes run the same checks.
</issue_to_address>

Sourcery is free for open source - if you like our reviews please consider sharing them ✨
Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.

@janisz janisz force-pushed the ROX-33792-undockerize-operator-build branch from 41e8b4b to 65aada2 Compare March 27, 2026 13:40
@rhacs-bot
Copy link
Copy Markdown
Contributor

rhacs-bot commented Mar 27, 2026

Images are ready for the commit at 9cd0473.

To use with deploy scripts, first export MAIN_IMAGE_TAG=4.11.x-496-g9cd047324b.

@janisz janisz force-pushed the ROX-33792-undockerize-operator-build branch from 65aada2 to 33aae4a Compare March 27, 2026 14:35
Build operator binary once in pre-build-go-binaries job alongside other
Go binaries, then reuse it when building Docker images. This eliminates
duplicate compilation and leverages GitHub Actions' Go module cache.

Changes:
- Add operator/cmd to main-build-nodeps in root Makefile
- Create operator/prebuilt.Dockerfile for CI builds
- Add conditional docker-build targets in operator/Makefile that use
  prebuilt binaries in CI mode, fall back to in-Docker build locally
- Allow operator binaries in .dockerignore and .containerignore
- Update build-and-push-operator job to download prebuilt binaries
  and skip proto resolution/duplicate builds

Benefits:
- ~2-5 min savings from eliminating duplicate Go compilation
- Better cache utilization via GitHub Actions' Go module cache
- Smaller Docker build context
- Fully backwards compatible (local builds unchanged)

Related: ROX-33792

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Signed-off-by: Tomasz Janiszewski <tomek@redhat.com>
@janisz janisz force-pushed the ROX-33792-undockerize-operator-build branch from 33aae4a to 7d9eafc Compare March 27, 2026 14:49
@codecov
Copy link
Copy Markdown

codecov bot commented Mar 27, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 49.64%. Comparing base (7072ab4) to head (9cd0473).
⚠️ Report is 2 commits behind head on master.

Additional details and impacted files
@@            Coverage Diff             @@
##           master   #19655      +/-   ##
==========================================
- Coverage   49.65%   49.64%   -0.01%     
==========================================
  Files        2747     2747              
  Lines      207261   207261              
==========================================
- Hits       102909   102902       -7     
- Misses      96698    96703       +5     
- Partials     7654     7656       +2     
Flag Coverage Δ
go-unit-tests 49.64% <ø> (-0.01%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Copy link
Copy Markdown
Contributor

@davdhacs davdhacs left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great. My only request is to remove the stackrox_operator exclude and put it into a .dockerignore that is only used for the prebuilt-operator.

Signed-off-by: Tomasz Janiszewski <tomek@redhat.com>
@janisz janisz force-pushed the ROX-33792-undockerize-operator-build branch from 0421a7f to fabb46b Compare March 30, 2026 08:25
Copy link
Copy Markdown
Contributor

@davdhacs davdhacs left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🥳

@janisz
Copy link
Copy Markdown
Contributor Author

janisz commented Mar 30, 2026

/retest

Copy link
Copy Markdown
Contributor

@porridge porridge left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

❤️

Co-authored-by: Marcin Owsiany <porridge@redhat.com>
@coderabbitai
Copy link
Copy Markdown
Contributor

coderabbitai bot commented Mar 31, 2026

📝 Walkthrough

Summary by CodeRabbit

  • Chores
    • Operator image build now uses prebuilt binaries, shortening CI build times and improving reliability.
    • Build workflow updated to fetch prebuilt artifacts before creating operator images.
    • Image build now supports explicit target-architecture builds and a streamlined prebuilt-image build path.

Walkthrough

The PR decouples operator binary compilation from image creation: a pre-build job produces prebuilt Go binaries, and the operator image job downloads those artifacts and builds the Docker image using a new prebuilt-aware Dockerfile/Make target.

Changes

Cohort / File(s) Summary
CI/CD Workflow
​.github/workflows/build.yaml
build-and-push-operator now depends on pre-build-go-binaries, downloads prebuilt Go binaries, sets ROX_OPERATOR_SKIP_PROTO_GENERATED_SRCS=true, and uses make -C operator/ docker-build-prebuilt instead of building Go/protos in-job.
Root Makefile
Makefile
main-build-nodeps now builds operator/cmd and moves/renames the produced binary to bin/linux_$(GOARCH)/stackrox-operator.
Operator Makefile
operator/Makefile
Added .PHONY target docker-build-prebuilt which invokes ../scripts/docker-build.sh with prebuilt.Dockerfile and forwards ROX_IMAGE_FLAVOR and optional GOARCH/TARGET_ARCH.
Operator Dockerfile & ignore
operator/prebuilt.Dockerfile, operator/prebuilt.Dockerfile.dockerignore
Added prebuilt.Dockerfile that copies bin/linux_${TARGET_ARCH}/stackrox-operator into the image (UBI9 micro base, runs as numeric nobody), and dockerignore to exclude everything except the prebuilt operator binary path.

Sequence Diagram(s)

mermaid
sequenceDiagram
participant PreBuild as Pre-build job
participant ArtifactStore as Artifact storage
participant OperatorJob as build-and-push-operator
participant DockerBuild as Docker build
PreBuild->>ArtifactStore: upload prebuilt Go binaries (tgz)
OperatorJob->>ArtifactStore: download artifact (go-binaries-build--default)
OperatorJob->>OperatorJob: extract binaries; set GOARCH/TARGET_ARCH
OperatorJob->>DockerBuild: invoke make docker-build-prebuilt (prebuilt.Dockerfile)
DockerBuild->>OperatorJob: produce operator image; push to registry

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~30 minutes

🚥 Pre-merge checks | ✅ 3
✅ Passed checks (3 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly and specifically describes the main change: building the operator binary outside Docker to improve build performance.
Description check ✅ Passed The PR description explains the key changes and rationale, though it lacks specific validation details beyond 'CI' and doesn't clearly address all template sections.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch ROX-33792-undockerize-operator-build

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

♻️ Duplicate comments (1)
operator/prebuilt.Dockerfile.dockerignore (1)

1-1: ⚠️ Potential issue | 🟠 Major

This Dockerfile-specific ignore file turns the ignore rules into a no-op.

For docker build -f prebuilt.Dockerfile .., Docker uses prebuilt.Dockerfile.dockerignore as the active ignore file rather than layering it on top of the repo-root .dockerignore. With only a ! rule here, nothing is excluded, so the whole repo gets sent as build context. That undercuts the CI speedup this PR is trying to get and can re-include paths the root ignore file normally keeps out. Please mirror the root ignore rules here (or drop the Dockerfile-specific file) before whitelisting bin/linux_*/stackrox-operator.

When both a root `.dockerignore` and a Dockerfile-specific file like `prebuilt.Dockerfile.dockerignore` exist, does Docker merge them or does the Dockerfile-specific file take precedence?
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@operator/prebuilt.Dockerfile.dockerignore` at line 1, The
prebuilt.Dockerfile.dockerignore currently contains only the whitelist rule
`!/bin/linux_*/stackrox-operator`, which makes the file override the root
.dockerignore and sends the entire repo as build context; update
prebuilt.Dockerfile.dockerignore to either mirror the root .dockerignore
contents (copy all root ignore rules into this file) and then append the
whitelist line `!/bin/linux_*/stackrox-operator`, or remove
prebuilt.Dockerfile.dockerignore entirely so Docker uses the root .dockerignore;
ensure the final file includes the same exclusions as the repository root before
the whitelist is applied.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In @.github/workflows/build.yaml:
- Around line 628-630: The operator job currently lists "needs:
pre-build-go-binaries" which forces it to wait for every matrix run of that job;
instead split or scope the dependency so the operator only waits for the
default-variant artifacts. Create a new job (e.g.,
pre-build-go-binaries-default) that only builds the default variant (producing
artifacts named like go-binaries-build-${{ matrix.arch }}-default) and change
the operator job's needs to reference that new job (replace needs:
pre-build-go-binaries with needs: pre-build-go-binaries-default), or
alternatively adjust the workflow to publish/select only the default-variant
artifact and have the operator depend on that specific job/output. Ensure the
operator continues to depend on define-job-matrix as before.

---

Duplicate comments:
In `@operator/prebuilt.Dockerfile.dockerignore`:
- Line 1: The prebuilt.Dockerfile.dockerignore currently contains only the
whitelist rule `!/bin/linux_*/stackrox-operator`, which makes the file override
the root .dockerignore and sends the entire repo as build context; update
prebuilt.Dockerfile.dockerignore to either mirror the root .dockerignore
contents (copy all root ignore rules into this file) and then append the
whitelist line `!/bin/linux_*/stackrox-operator`, or remove
prebuilt.Dockerfile.dockerignore entirely so Docker uses the root .dockerignore;
ensure the final file includes the same exclusions as the repository root before
the whitelist is applied.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Central YAML (base), Organization UI (inherited)

Review profile: CHILL

Plan: Pro

Run ID: f848051b-2bc3-4a25-8444-ad2a36d5f7bd

📥 Commits

Reviewing files that changed from the base of the PR and between 0737966 and 24ecc6c.

📒 Files selected for processing (5)
  • .github/workflows/build.yaml
  • Makefile
  • operator/Makefile
  • operator/prebuilt.Dockerfile
  • operator/prebuilt.Dockerfile.dockerignore

@janisz janisz added the auto-retest PRs with this label will be automatically retested if prow checks fails label Mar 31, 2026
Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (1)
.github/workflows/build.yaml (1)

674-678: Consider adding if: ${{ !cancelled() }} for consistency with build-and-push-main.

The build-and-push-main job (line 387) uses if: ${{ !cancelled() }} to allow partial matrix completion when some prerequisite jobs fail. This enables building images for architectures where prebuilds succeeded, even if others failed.

The build-and-push-operator job lacks this condition, meaning it will be entirely skipped if any pre-build-go-binaries matrix run fails—even for architectures where prebuilds succeeded.

If this asymmetry is intentional (e.g., stricter requirements for operator builds), this comment can be disregarded.

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In @.github/workflows/build.yaml around lines 674 - 678, The
build-and-push-operator job lacks the cancellation guard present in
build-and-push-main; update the build-and-push-operator job declaration to
include the same condition by adding if: ${{ !cancelled() }} so the operator job
can proceed for matrix entries that completed even if other pre-build matrix
runs failed (apply this change to the job named build-and-push-operator to
mirror build-and-push-main).
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Nitpick comments:
In @.github/workflows/build.yaml:
- Around line 674-678: The build-and-push-operator job lacks the cancellation
guard present in build-and-push-main; update the build-and-push-operator job
declaration to include the same condition by adding if: ${{ !cancelled() }} so
the operator job can proceed for matrix entries that completed even if other
pre-build matrix runs failed (apply this change to the job named
build-and-push-operator to mirror build-and-push-main).

ℹ️ Review info
⚙️ Run configuration

Configuration used: Central YAML (base), Organization UI (inherited)

Review profile: CHILL

Plan: Pro

Run ID: f4959d42-82df-4260-9400-3c716669ddeb

📥 Commits

Reviewing files that changed from the base of the PR and between 24ecc6c and 9cd0473.

📒 Files selected for processing (2)
  • .github/workflows/build.yaml
  • operator/Makefile
✅ Files skipped from review due to trivial changes (1)
  • operator/Makefile

@janisz janisz merged commit 416d100 into master Mar 31, 2026
114 of 115 checks passed
@janisz janisz deleted the ROX-33792-undockerize-operator-build branch March 31, 2026 13:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ai-review area/ci area/operator auto-retest PRs with this label will be automatically retested if prow checks fails coderabbit-review

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants