perf(ci): parallelize style-slim via dynamic matrix#19312
Draft
perf(ci): parallelize style-slim via dynamic matrix#19312
Conversation
Remove the container: directive from all 4 container-based jobs in the Style workflow (check-generated-files, misc-checks, style-check, openshift-ci-lint) and run them directly on the ubuntu-latest runner. Container initialization consistently takes ~55s per job. Replacing it with on-host tool setup (setup-go, setup-node, yq, xmlstarlet, pip) costs ~5-15s, saving ~40-50s per job. Most tools (Go, Node.js, Make, GCC, Git, Helm, shellcheck, jq) are already preinstalled on the runner. For style-check, the shell-based tool installations (yq, xmlstarlet, pip) run as parallel background subshells to minimize wall-clock time. Also updates cache-ui-dependencies to use portable ~/. paths instead of container-specific /github/home/ paths, with a cache key bump (v2->v3). Generated with the assistance of AI. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Address sourcery-ai review feedback: - Add SHA256 checksum verification after downloading yq binary to mitigate supply-chain risk - Use `python3 -m pip` instead of bare `pip` for deterministic Python interpreter resolution across runner image updates Generated with the assistance of AI. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
actions/setup-go sets GOTOOLCHAIN=local by default, which prevents Go from auto-downloading a newer toolchain when a dependency (e.g. buf) requires one. The CI container didn't have this restriction. Set GOTOOLCHAIN=auto at the workflow level so Go tools that require a newer patch version (e.g. buf needing go1.25.6 when go.mod says 1.25.0) can download the required toolchain automatically. Generated with the assistance of AI. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
actions/setup-go writes GOTOOLCHAIN=local to GITHUB_ENV, which overrides any workflow-level env setting. This prevents Go tools in sub-modules with newer go.mod directives (e.g. tools/proto requires go 1.25.6) from auto-downloading the required toolchain. Write GOTOOLCHAIN=auto to GITHUB_ENV in a step after setup-go so it takes effect in all subsequent steps. This matches the behavior in the CI container where GOTOOLCHAIN was unset (defaulting to auto). The project uses GOTOOLCHAIN=local only in targeted make targets (go mod tidy) as a compatibility guard — not as a global setting. Generated with the assistance of AI. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The operator's bundle_helpers virtualenv uses packages (pip, pytest) that rely on pkgutil.ImpImporter, which was removed in Python 3.12. The CI container had Python 3.9 (UBI8) where this worked. The ubuntu-latest runner has Python 3.12. Add actions/setup-python with python-version 3.9 to match the container's Python version until operator/bundle_helpers is updated for Python 3.12 compatibility. Generated with the assistance of AI. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Replace the monolithic style-check job (which ran ~11 make targets sequentially in ~26 min) with a dynamic matrix that: 1. A lightweight discover job parses style-slim prerequisites from the Makefile, so adding/removing targets in the Makefile automatically updates the CI matrix without workflow changes. 2. Each target runs as an independent parallel matrix cell with conditional setup — only ui-lint gets Node.js, only shell-style gets xmlstarlet, only openshift-ci-style gets Python linters, etc. Expected wall-clock improvement: from ~26 min (sum of all targets) to ~15 min (longest single target, likely roxvet) plus ~2 min setup. Generated with the assistance of AI. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
|
Skipping CI for Draft Pull Request. |
![sourcery-ai[bot]](https://avatars.githubusercontent.com/in/48477?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Hey - I've left some high level feedback:
- The Makefile parsing in the
style-targetsjob (sed+tr+xargs+sed) is quite brittle against formatting changes (e.g., comments, reordering, extra whitespace); consider tightening this (e.g., anchoring on a dedicated variable or pattern, or adding a small script with clearer parsing logic) so a benign Makefile edit doesn’t silently skew the matrix. - The
yqinstallation logic is duplicated in multiple jobs with nearly identical shell blocks; consider extracting this into a small composite action or reusable step to avoid divergence when updating versions or flags later.
Prompt for AI Agents
Please address the comments from this code review:
## Overall Comments
- The Makefile parsing in the `style-targets` job (`sed` + `tr` + `xargs` + `sed`) is quite brittle against formatting changes (e.g., comments, reordering, extra whitespace); consider tightening this (e.g., anchoring on a dedicated variable or pattern, or adding a small script with clearer parsing logic) so a benign Makefile edit doesn’t silently skew the matrix.
- The `yq` installation logic is duplicated in multiple jobs with nearly identical shell blocks; consider extracting this into a small composite action or reusable step to avoid divergence when updating versions or flags later.Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.
Contributor
|
Images are ready for the commit at 9e3f959. To use with deploy scripts, first |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #19312 +/- ##
=======================================
Coverage 49.59% 49.60%
=======================================
Files 2756 2756
Lines 208036 208036
=======================================
+ Hits 103183 103190 +7
+ Misses 97192 97187 -5
+ Partials 7661 7659 -2
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
actionlint runs shellcheck on inline scripts and flags the backslash in tr '\\' ' ' as SC1003. This is a false positive — the double backslash is intentional (tr needs a literal backslash argument). Generated with the assistance of AI. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
roxvet is the bottleneck in the style-check matrix at ~18 min, running go vet with a custom analyzer over ~1946 packages twice (with/without extra build tags). Add ROXVET_SCOPE variable to the Makefile (defaults to ./...) so CI can scope roxvet to a subset of packages. The workflow's discover step expands 'roxvet' into 3 shards by top-level directory: - roxvet:central (966 packages, ~9 min) - roxvet:pkg (474 packages, ~5 min) - roxvet:other (~506 packages, ~5 min) Expected style-check wall clock: ~9-10 min (central shard) instead of ~18 min (full roxvet). Generated with the assistance of AI. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Replace sed with ${var/pattern/replacement} to satisfy shellcheck
SC2001 (flagged by actionlint).
Generated with the assistance of AI.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Address review feedback from janisz: - Remove yq wget/checksum from check-generated-files and misc-checks: yq is preinstalled on ubuntu-24.04 runners (v4.52.4). - Separate the GOTOOLCHAIN override into its own step with a comment explaining why it's needed (setup-go@v6 forces GOTOOLCHAIN=local). Generated with the assistance of AI. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Move the Python version constraint next to the code that causes it (pinned pip==21.3.1 and setuptools==59.6.0 in operator/Makefile). The workflow now reads from the .python-version file instead of hardcoding the version.
Replace the monolithic style-check job (~17m) with 4 parallel jobs split by dependency type: - style-go-checks: roxvet, proto-style, check-service-protos, storage-protos-compatible (needs Go) - style-ui-lint: ui-lint (needs Node) - style-qa-tests: qa-tests-style (needs Go + Gradle) - style-shell-checks: blanks, newlines, no-large-files, shell-style, openshift-ci-style (needs shellcheck + pycodestyle) Each job installs only the tools it needs instead of all of them. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Contributor
📝 WalkthroughWalkthroughThese changes refactor the CI workflow's monolithic style-check job into four parallelized, scoped jobs targeting Go, UI, QA, and shell code quality checks. The Makefile introduces a parameterizable Changes
Estimated code review effort🎯 2 (Simple) | ⏱️ ~10 minutes 🚥 Pre-merge checks | ✅ 3✅ Passed checks (3 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In @.github/workflows/style.yaml:
- Around line 156-158: The style-ui-lint job's checkout step currently uses
actions/checkout@v6 without a ref, causing it to check out the merge commit;
update the checkout step in the style-ui-lint job (job name "style-ui-lint") to
pass ref: ${{ github.event.pull_request.head.sha }} to actions/checkout@v6 so
the job checks out the actual PR head commit (matching how style-go-checks,
style-qa-tests, and style-shell-checks are configured).
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Central YAML (base), Organization UI (inherited)
Review profile: CHILL
Plan: Pro
Run ID: a4c3746b-2d68-4d0d-83fe-ceb4fcdaf1b6
📒 Files selected for processing (2)
.github/workflows/style.yamlMakefile
Comment on lines
+156
to
+158
| - name: Checkout | ||
| uses: actions/checkout@v6 | ||
|
|
Contributor
There was a problem hiding this comment.
Inconsistent checkout configuration with other style jobs.
The style-ui-lint job doesn't specify ref: ${{ github.event.pull_request.head.sha }} in its checkout step, unlike the other new style jobs (style-go-checks, style-qa-tests, style-shell-checks). On PRs, this causes the job to check out GitHub's merge commit rather than the actual PR head, potentially leading to inconsistent results.
Proposed fix
- name: Checkout
uses: actions/checkout@v6
+ with:
+ ref: ${{ github.event.pull_request.head.sha }}📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| - name: Checkout | |
| uses: actions/checkout@v6 | |
| - name: Checkout | |
| uses: actions/checkout@v6 | |
| with: | |
| ref: ${{ github.event.pull_request.head.sha }} |
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In @.github/workflows/style.yaml around lines 156 - 158, The style-ui-lint job's
checkout step currently uses actions/checkout@v6 without a ref, causing it to
check out the merge commit; update the checkout step in the style-ui-lint job
(job name "style-ui-lint") to pass ref: ${{ github.event.pull_request.head.sha
}} to actions/checkout@v6 so the job checks out the actual PR head commit
(matching how style-go-checks, style-qa-tests, and style-shell-checks are
configured).
qa-tests-style needs Go (for copy_scanner_protos.sh → go list -m) and GOMODCACHE (for scanner module resolution). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
roxvet runs go vet twice across ~2000 packages (~15m). Split into 3 shards by top-level directory: - central: 983 packages (50%) - pkg: 475 packages (24%) - rest: 516 packages (26%) Each shard runs in parallel → ~5-6m instead of ~15m. Also separate proto-style/check-service-protos into own job. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Contributor
|
CodeRabbit chat interactions are restricted to organization members for this repository. Ask an organization member to interact with CodeRabbit, or set |
# Conflicts: # .github/workflows/style.yaml
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Split the monolithic
style-checkjob into parallel jobs by dependency type, and shard roxvet across 3 package groups.Master: style-check 19.8m → wall-clock 13m (34% faster).
Before (monolithic)
After (parallelized)
Each job installs only the tools it needs (Go, Node, Gradle, shellcheck) instead of all of them.
User-facing documentation
Testing and quality
Automated testing
How I validated my change
All 7 new style jobs pass. qa-tests and proto-checks verified with Go cache for scanner module resolution.
Partially generated by AI.