Skip to content

perf(ci): parallelize style-slim via dynamic matrix #19312

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
davdhacs wants to merge 23 commits into
base: master
Choose a base branch
from
Draft

perf(ci): parallelize style-slim via dynamic matrix #19312

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
23 commits
Select commit Hold shift + click to select a range
8539333
perf(ci): remove container from style.yaml jobs, run on host
davdhacs Mar 5, 2026
3c0d44c
ci: trigger workflow run
davdhacs Mar 5, 2026
aff6277
ci: re-trigger workflow
davdhacs Mar 5, 2026
a772437
fix(ci): verify yq checksum and use python3 -m pip
davdhacs Mar 5, 2026
0afebf9
fix(ci): set GOTOOLCHAIN=auto to allow toolchain downloads
davdhacs Mar 5, 2026
264d7df
fix(ci): override GOTOOLCHAIN=local after setup-go via GITHUB_ENV
davdhacs Mar 5, 2026
5bdb821
fix(ci): add setup-python 3.9 for check-generated-files
davdhacs Mar 5, 2026
d4fc172
perf(ci): parallelize style-slim via dynamic matrix
davdhacs Mar 6, 2026
9ae6147
fix(ci): suppress shellcheck SC1003 in style-targets discovery
davdhacs Mar 6, 2026
2260217
perf(ci): shard roxvet across 3 parallel matrix cells
davdhacs Mar 6, 2026
f958ab2
fix(ci): use bash parameter expansion for roxvet shard substitution
davdhacs Mar 6, 2026
cd27a71
fix(ci): drop yq install (preinstalled on runner) and clean up steps
davdhacs Mar 9, 2026
1d61ed4
fix(ci): use node-version-file instead of hardcoded node version
davdhacs Mar 12, 2026
7c0a97f
fix(ci): clarify Python 3.9 pin reason (operator Makefile pins pip 21…
davdhacs Mar 12, 2026
e2899f4
fix(ci): track Python version in operator/bundle_helpers/.python-version
davdhacs Mar 12, 2026
cf316b8
Merge origin/master into style-parallel-matrix
davdhacs Mar 31, 2026
e39a2c8
perf(ci): split style-check into 4 parallel jobs
davdhacs Mar 31, 2026
a7ec316
fix: add Go setup and cache to style-qa-tests job
davdhacs Mar 31, 2026
56f5d41
perf(ci): shard roxvet across 3 parallel jobs
davdhacs Mar 31, 2026
db40803
Merge remote-tracking branch 'origin/master' into style-parallel-matrix
davdhacs Mar 31, 2026
2867bf9
Merge remote-tracking branch 'origin/master' into style-parallel-matrix
davdhacs Mar 31, 2026
cd7e94d
Merge remote-tracking branch 'origin/master' into style-parallel-matrix
davdhacs Mar 31, 2026
9e3f959
Merge remote-tracking branch 'origin/master' into style-parallel-matrix
davdhacs Apr 1, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
150 changes: 126 additions & 24 deletions .github/workflows/style.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -114,7 +114,21 @@ jobs:
- name: Check dependabot gomod configurations match go.mod files
run: if [ -s scripts/ci/jobs/check-dependabot-gomod.sh ]; then scripts/ci/jobs/check-dependabot-gomod.sh; fi

style-check:
# style-slim sub-targets split by dependency type for parallelism.
# The monolithic style-check took ~17m running all sub-targets serially.
# roxvet (~15m) is further sharded by package scope.

style-roxvet:
strategy:
fail-fast: false
matrix:
shard:
- name: central
scope: ./central/...
- name: pkg
scope: ./pkg/...
- name: rest
scope: ./sensor/... ./roxctl/... ./tools/... ./migrator/... ./operator/... ./scanner/... ./compliance/... ./generated/... ./config-controller/... ./scale/... ./govulncheck/... ./image/...
runs-on: ubuntu-latest
steps:
- name: Checkout
Expand All @@ -125,54 +139,138 @@ jobs:

- uses: ./.github/actions/job-preamble
with:
free-disk-space: '30'
gcp-account: ${{ secrets.GCP_SERVICE_ACCOUNT_STACKROX_CI }}

- uses: actions/setup-go@v6
with:
go-version-file: go.mod
cache: false

# setup-go@v6 exports GOTOOLCHAIN=local; override so sub-module tools
# with newer go directives (e.g. tools/proto) can auto-download.
- name: Override GOTOOLCHAIN
run: echo "GOTOOLCHAIN=auto" >> "$GITHUB_ENV"

- uses: actions/setup-node@v6
- name: Cache Go dependencies
uses: ./.github/actions/cache-go-dependencies

- name: roxvet
run: ROXVET_SCOPE="${{ matrix.shard.scope }}" make roxvet

style-proto-checks:
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v6
with:
node-version-file: ui/package.json
fetch-depth: 0
ref: ${{ github.event.pull_request.head.sha }}

- name: Install xmlstarlet for shellcheck output to junit
run: |
sudo apt-get update -qq && sudo apt-get install -y -qq xmlstarlet
- uses: ./.github/actions/job-preamble
with:
gcp-account: ${{ secrets.GCP_SERVICE_ACCOUNT_STACKROX_CI }}

- name: Install pycodestyle
run: |
python3 -m pip install -r .openshift-ci/dev-requirements.txt
- uses: actions/setup-go@v6
with:
go-version-file: go.mod
cache: false

- name: Verify tools
run: |
yq --version
xmlstarlet --version
pycodestyle --version
- name: Override GOTOOLCHAIN
run: echo "GOTOOLCHAIN=auto" >> "$GITHUB_ENV"

- name: Cache Go dependencies
uses: ./.github/actions/cache-go-dependencies

- name: Download scanner module for proto generation
run: go mod download github.com/stackrox/scanner
- name: proto-style
run: make proto-style

- name: check-service-protos
run: make check-service-protos

- name: storage-protos-compatible
run: make storage-protos-compatible

style-ui-lint:
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v6

Comment on lines +194 to +196
Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai bot Mar 31, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

Inconsistent checkout configuration with other style jobs.

The style-ui-lint job doesn't specify ref: ${{ github.event.pull_request.head.sha }} in its checkout step, unlike the other new style jobs (style-go-checks, style-qa-tests, style-shell-checks). On PRs, this causes the job to check out GitHub's merge commit rather than the actual PR head, potentially leading to inconsistent results.

Proposed fix 
       - name: Checkout
         uses: actions/checkout@v6
+        with:
+          ref: ${{ github.event.pull_request.head.sha }}
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
- name: Checkout
uses: actions/checkout@v6
- name: Checkout
uses: actions/checkout@v6
with:
ref: ${{ github.event.pull_request.head.sha }}
🤖 Prompt for AI Agents 
Verify each finding against the current code and only fix it if needed.

In @.github/workflows/style.yaml around lines 156 - 158, The style-ui-lint job's
checkout step currently uses actions/checkout@v6 without a ref, causing it to
check out the merge commit; update the checkout step in the style-ui-lint job
(job name "style-ui-lint") to pass ref: ${{ github.event.pull_request.head.sha
}} to actions/checkout@v6 so the job checks out the actual PR head commit
(matching how style-go-checks, style-qa-tests, and style-shell-checks are
configured).

- uses: ./.github/actions/job-preamble
with:
gcp-account: ${{ secrets.GCP_SERVICE_ACCOUNT_STACKROX_CI }}

- uses: actions/setup-node@v6
with:
node-version-file: ui/package.json

- name: Cache UI dependencies
uses: ./.github/actions/cache-ui-dependencies

- name: Fetch UI deps
run: make -C ui deps

- name: ui-lint
run: make ui-lint

style-qa-tests:
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v6
with:
fetch-depth: 0
ref: ${{ github.event.pull_request.head.sha }}

- uses: ./.github/actions/job-preamble
with:
gcp-account: ${{ secrets.GCP_SERVICE_ACCOUNT_STACKROX_CI }}

- uses: actions/setup-go@v6
with:
go-version-file: go.mod
cache: false

- name: Cache Go dependencies
uses: ./.github/actions/cache-go-dependencies

- name: Cache QA Test dependencies
uses: ./.github/actions/cache-gradle-dependencies

- name: Fetch UI deps
run: make -C ui deps
- name: qa-tests-style
run: make qa-tests-style

style-shell-checks:
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v6
with:
fetch-depth: 0
ref: ${{ github.event.pull_request.head.sha }}

- uses: ./.github/actions/job-preamble
with:
gcp-account: ${{ secrets.GCP_SERVICE_ACCOUNT_STACKROX_CI }}

- name: Install xmlstarlet
run: sudo apt-get update -qq && sudo apt-get install -y -qq xmlstarlet

- name: Install pycodestyle
run: python3 -m pip install -r .openshift-ci/dev-requirements.txt

- name: blanks
run: make blanks

- name: newlines
run: make newlines

- name: no-large-files
run: make no-large-files

- name: shell-style
run: make shell-style

- name: make style-slim
run: make style-slim
- name: openshift-ci-style
run: make openshift-ci-style

golangci-lint:
timeout-minutes: 240
Expand Down Expand Up @@ -339,7 +437,11 @@ jobs:
needs:
- check-generated-files
- misc-checks
- style-check
- style-roxvet
- style-proto-checks
- style-ui-lint
- style-qa-tests
- style-shell-checks
- golangci-lint
- check-dependent-images-exist
- github-actions-lint
Expand Down
6 changes: 4 additions & 2 deletions Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -811,15 +811,17 @@ reinstall-dev-tools: clean-dev-tools
install-dev-tools: gotools-all
@echo "+ $@"

ROXVET_SCOPE ?= ./...

.PHONY: roxvet
roxvet: skip-dirs := operator/pkg/clientset
roxvet: $(ROXVET_BIN)
@echo "+ $@"
@# TODO(ROX-7574): Add options to ignore specific files or paths in roxvet
$(SILENT)go list -e ./... \
$(SILENT)go list -e $(ROXVET_SCOPE) \
| $(foreach d,$(skip-dirs),grep -v '$(d)' |) \
xargs -n 1000 go vet -vettool "$(ROXVET_BIN)" -donotcompareproto -gogoprotofunctions -tags "sql_integration test_e2e test race destructive integration scanner_db_integration compliance externalbackups"
$(SILENT)go list -e ./... \
$(SILENT)go list -e $(ROXVET_SCOPE) \
| $(foreach d,$(skip-dirs),grep -v '$(d)' |) \
xargs -n 1000 go vet -vettool "$(ROXVET_BIN)"

Expand Down
Loading