Skip to content

perf(ci): shard go and go-postgres unit test jobs #19744

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
davdhacs wants to merge 18 commits into
base: master
Choose a base branch
from
Draft

perf(ci): shard go and go-postgres unit test jobs #19744

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
18 commits
Select commit Hold shift + click to select a range
b9c014f
fix: host-runner compatibility for tests and scripts
davdhacs Mar 31, 2026
3f50506
Merge remote-tracking branch 'origin/master' into davdhacs/host-runne…
davdhacs Mar 31, 2026
fe90602
Merge remote-tracking branch 'origin/master' into davdhacs/host-runne…
davdhacs Apr 1, 2026
af1275f
perf(ci): add SKIP_DEPS, split postgres targets, allow test caching
davdhacs Apr 1, 2026
58ea483
fix: use sed instead of grep -v in yq_multidoc
davdhacs Apr 1, 2026
f0fa0ff
perf(ci): remove container from go, go-postgres, go-bench jobs
davdhacs Apr 1, 2026
928766b
perf(ci): shard go and go-postgres unit test jobs
davdhacs Apr 1, 2026
aaaff23
Merge remote-tracking branch 'origin/master' into davdhacs/shard-go-t…
davdhacs Apr 1, 2026
fc6eb71
Merge remote-tracking branch 'origin/davdhacs/makefile-skip-deps' int…
davdhacs Apr 1, 2026
9ea6507
refactor: use exclude pattern instead of hardcoded package list for r…
davdhacs Apr 1, 2026
220aa58
fix: use consistent BUILD_TAG/SHORTCOMMIT on go-operator-integration
davdhacs Apr 1, 2026
62d8bca
fix(ci): download scanner module for operator integration tests
davdhacs Apr 1, 2026
44300dd
fix(ci): remove junit steps from go-operator-integration
davdhacs Apr 1, 2026
d37671b
docs(ci): add GOTOOLCHAIN override comments to all jobs
davdhacs Apr 1, 2026
298cc3c
docs(ci): clarify GOTOOLCHAIN override comments
davdhacs Apr 1, 2026
b12da02
docs(ci): fix GOTOOLCHAIN override comments
davdhacs Apr 1, 2026
85f29e3
docs(ci): explain postgres docker config
davdhacs Apr 1, 2026
9c2e078
fix(ci): remove go-bench skip from shard PR
davdhacs Apr 1, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
244 changes: 190 additions & 54 deletions .github/workflows/unit-tests.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,21 @@ jobs:
fail-fast: false
matrix:
gotags: [ 'GOTAGS=""', 'GOTAGS=release' ]
shard:
- name: pkg-helm
packages: ./pkg/helm/...
- name: pkg-other
packages: ./pkg/...
exclude-pattern: /helm/
- name: central-1
packages: ./central/...
exclude-pattern: /central/[o-z]
- name: central-2
packages: ./central/...
exclude-pattern: /central/[a-n]
- name: rest
packages: ./...
exclude-pattern: /pkg/|/central/
runs-on: ubuntu-latest
outputs:
new-jiras: ${{ steps.junit2jira.outputs.new-jiras }}
Expand All @@ -31,16 +46,20 @@ jobs:
env:
BUILD_TAG: 0.0.0
SHORTCOMMIT: "0000000"
container:
image: quay.io/stackrox-io/apollo-ci:stackrox-test-0.5.3
volumes:
- /usr:/mnt/usr
- /opt:/mnt/opt
steps:
- name: Checkout
uses: actions/checkout@v6

- uses: actions/setup-go@v6
with:
fetch-depth: 0
go-version-file: go.mod
cache: false

# setup-go exports GOTOOLCHAIN matching the go.mod value, which
# prevents auto-downloading newer toolchains. Override to auto so
# sub-modules with newer go directives (e.g. tools/proto) work.
- name: Override GOTOOLCHAIN
run: echo "GOTOOLCHAIN=auto" >> "$GITHUB_ENV"

- uses: ./.github/actions/job-preamble
with:
Expand All @@ -50,15 +69,25 @@ jobs:
- name: Cache Go dependencies
uses: ./.github/actions/cache-go-dependencies
with:
key-suffix: ${{ matrix.gotags }}
key-suffix: ${{ matrix.gotags }}-${{ matrix.shard.name }}

- name: Go Unit Tests
run: ${{ matrix.gotags }} make go-unit-tests

- uses: codecov/codecov-action@v3
with:
token: ${{ secrets.CODECOV_TOKEN }}
flags: go-unit-tests
run: |
mkdir -p bin test-output
set -o pipefail
# Build package list, filtering out packages that have their own dedicated jobs
EXCLUDE='sensor/tests|operator/tests|/scanner/e2etests'
if [[ -n '${{ matrix.shard.exclude-pattern }}' ]]; then
EXCLUDE="${EXCLUDE}|${{ matrix.shard.exclude-pattern }}"
fi
PACKAGES=$(go list ${{ matrix.shard.packages }} | grep -Ev "$EXCLUDE")
# shellcheck disable=SC2086
GOTAGS="${GOTAGS:+$GOTAGS,}test" CGO_ENABLED=1 GOEXPERIMENT=cgocheck2 MUTEX_WATCHDOG_TIMEOUT_SECS=30 \
scripts/go-test.sh -timeout 25m -race -cover \
-coverprofile test-output/coverage.out -v \
$PACKAGES \
| tee test-output/test.log
Comment on lines +85 to +89
Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai bot Apr 1, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🔴 Critical

Missing ${{ matrix.gotags }} causes both matrix variations to run identical tests.

The gotags matrix dimension is defined (line 24) but not used in this command. Both GOTAGS="" and GOTAGS=release matrix jobs will execute with identical GOTAGS=test, defeating the purpose of testing with release tags.

Compare with go-integration job (line 147) which correctly uses ${{ matrix.gotags }} prefix.

🐛 Proposed fix 
     # shellcheck disable=SC2086
-    GOTAGS="${GOTAGS:+$GOTAGS,}test" CGO_ENABLED=1 GOEXPERIMENT=cgocheck2 MUTEX_WATCHDOG_TIMEOUT_SECS=30 \
+    ${{ matrix.gotags }} GOTAGS="${GOTAGS:+$GOTAGS,}test" CGO_ENABLED=1 GOEXPERIMENT=cgocheck2 MUTEX_WATCHDOG_TIMEOUT_SECS=30 \
       scripts/go-test.sh -timeout 25m -race -cover \
🤖 Prompt for AI Agents 
Verify each finding against the current code and only fix it if needed.

In @.github/workflows/unit-tests.yaml around lines 83 - 87, The unit-tests job
is always setting GOTAGS to "test" and ignores the matrix dimension; update the
GOTAGS environment passed to scripts/go-test.sh so it includes the matrix value
(`${{ matrix.gotags }}`) as a prefix before the existing test tag (so the matrix
variations actually run with different GOTAGS), i.e. change the GOTAGS
assignment used when invoking go-test.sh (the GOTAGS=... before
scripts/go-test.sh) to incorporate `${{ matrix.gotags }}` while preserving the
existing ",test" behavior.

shell: bash

- name: Generate junit report
if: always()
Expand All @@ -70,14 +99,59 @@ jobs:
with:
paths: 'junit-reports/report.xml'

- name: Go Integration Unit Tests
run: ${{ matrix.gotags }} make integration-unit-tests
- name: Report test failures to Jira
if: (!cancelled())
id: junit2jira
uses: ./.github/actions/junit2jira
with:
create-jiras: ${{ github.event_name == 'push' }}
jira-user: ${{ secrets.JIRA_USER }}
jira-token: ${{ secrets.JIRA_TOKEN }}
gcp-account: ${{ secrets.GCP_SERVICE_ACCOUNT_STACKROX_CI }}
directory: 'junit-reports'

- name: Go Operator Integration Tests
run: ${{ matrix.gotags }} make -C operator/ test-integration
go-integration:
strategy:
fail-fast: false
matrix:
gotags: [ 'GOTAGS=""', 'GOTAGS=release' ]
runs-on: ubuntu-latest
env:
BUILD_TAG: 0.0.0
SHORTCOMMIT: "0000000"
steps:
- name: Checkout
uses: actions/checkout@v6

- name: Go Operator Helm Tests
run: ${{ matrix.gotags }} make -C operator/ test-helm
- uses: actions/setup-go@v6
with:
go-version-file: go.mod
cache: false

# setup-go exports GOTOOLCHAIN matching the go.mod value, which
# prevents auto-downloading newer toolchains. Override to auto so
# sub-modules with newer go directives (e.g. tools/proto) work.
- name: Override GOTOOLCHAIN
run: echo "GOTOOLCHAIN=auto" >> "$GITHUB_ENV"

- uses: ./.github/actions/job-preamble
with:
gcp-account: ${{ secrets.GCP_SERVICE_ACCOUNT_STACKROX_CI }}

- name: Cache Go dependencies
uses: ./.github/actions/cache-go-dependencies
with:
key-suffix: ${{ matrix.gotags }}-integration

- name: Go Integration Unit Tests
run: |
mkdir -p test-output
set -o pipefail
PACKAGES=$(go list ./... | grep -E 'registries|scanners|notifiers')
# shellcheck disable=SC2086
${{ matrix.gotags }} GOTAGS="${GOTAGS:+$GOTAGS,}test,integration" CGO_ENABLED=1 GOEXPERIMENT=cgocheck2 MUTEX_WATCHDOG_TIMEOUT_SECS=30 \
scripts/go-test.sh -v $PACKAGES | tee test-output/test.log
shell: bash

- name: Generate junit report
if: always()
Expand All @@ -89,23 +163,57 @@ jobs:
with:
paths: 'junit-reports/report.xml'

- name: Report test failures to Jira
if: (!cancelled())
id: junit2jira
uses: ./.github/actions/junit2jira
go-operator-integration:
strategy:
fail-fast: false
matrix:
gotags: [ 'GOTAGS=""', 'GOTAGS=release' ]
runs-on: ubuntu-latest
env:
BUILD_TAG: 0.0.0
SHORTCOMMIT: "0000000"
steps:
- name: Checkout
uses: actions/checkout@v6

- uses: actions/setup-go@v6
with:
go-version-file: go.mod
cache: false

# setup-go exports GOTOOLCHAIN matching the go.mod value, which
# prevents auto-downloading newer toolchains. Override to auto so
# sub-modules with newer go directives (e.g. tools/proto) work.
- name: Override GOTOOLCHAIN
run: echo "GOTOOLCHAIN=auto" >> "$GITHUB_ENV"

- uses: ./.github/actions/job-preamble
with:
create-jiras: ${{ github.event_name == 'push' }}
jira-user: ${{ secrets.JIRA_USER }}
jira-token: ${{ secrets.JIRA_TOKEN }}
gcp-account: ${{ secrets.GCP_SERVICE_ACCOUNT_STACKROX_CI }}
directory: 'junit-reports'

- name: Cache Go dependencies
uses: ./.github/actions/cache-go-dependencies
Comment on lines +194 to +195
Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai bot Apr 1, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

Missing key-suffix for cache differentiation between gotags matrix runs.

Other jobs include key-suffix: ${{ matrix.gotags }}-... to prevent cache collisions between matrix variations. This job omits it, which could cause cache conflicts.

🔧 Proposed fix 
     - name: Cache Go dependencies
       uses: ./.github/actions/cache-go-dependencies
+      with:
+        key-suffix: ${{ matrix.gotags }}-operator-integration
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
- name: Cache Go dependencies
uses: ./.github/actions/cache-go-dependencies
- name: Cache Go dependencies
uses: ./.github/actions/cache-go-dependencies
with:
key-suffix: ${{ matrix.gotags }}-operator-integration
🤖 Prompt for AI Agents 
Verify each finding against the current code and only fix it if needed.

In @.github/workflows/unit-tests.yaml around lines 185 - 186, The "Cache Go
dependencies" step using ./.github/actions/cache-go-dependencies is missing a
key-suffix to separate caches for different matrix runs; update that step to
include a key-suffix like ${{ matrix.gotags }} (or ${{ matrix.gotags
}}-<other-identifiers>) so the cache key differs per gotags matrix entry and
prevents collisions between matrix variations.


- name: Download scanner module for proto generation
run: go mod download github.com/stackrox/scanner

- name: Go Operator Integration Tests
run: ${{ matrix.gotags }} make -C operator/ test-integration

- name: Go Operator Helm Tests
run: ${{ matrix.gotags }} make -C operator/ test-helm

go-postgres:
strategy:
fail-fast: false
matrix:
gotags: [ 'GOTAGS=""', 'GOTAGS=release' ]
pg: [ '15' ]
shard:
- name: main
make-target: go-postgres-unit-tests-main
- name: migrator
make-target: go-postgres-unit-tests-migrator
runs-on: ubuntu-latest
outputs:
new-jiras: ${{ steps.junit2jira.outputs.new-jiras }}
Expand All @@ -115,40 +223,52 @@ jobs:
env:
BUILD_TAG: 0.0.0
SHORTCOMMIT: "0000000"
container:
image: quay.io/stackrox-io/apollo-ci:stackrox-test-0.5.3
volumes:
- /usr:/mnt/usr
- /opt:/mnt/opt
steps:
- name: Set Postgres version
run: |
echo "/usr/pgsql-${{ matrix.pg }}/bin" >> "${GITHUB_PATH}"

- name: Checkout
uses: actions/checkout@v6

- uses: actions/setup-go@v6
with:
fetch-depth: 0
go-version-file: go.mod
cache: false

# setup-go exports GOTOOLCHAIN matching the go.mod value, which
# prevents auto-downloading newer toolchains. Override to auto so
# sub-modules with newer go directives (e.g. tools/proto) work.
- name: Override GOTOOLCHAIN
run: echo "GOTOOLCHAIN=auto" >> "$GITHUB_ENV"

- uses: ./.github/actions/job-preamble
with:
gcp-account: ${{ secrets.GCP_SERVICE_ACCOUNT_STACKROX_CI }}

# trust: allow passwordless TCP connections (replaces unix socket peer auth from CI container)
# locale=C: deterministic collation matching CI container's default
- name: Run Postgres
run: |
su postgres -c 'initdb -D /tmp/data'
su postgres -c 'pg_ctl -D /tmp/data start'
docker run --rm -d --name postgres \
-e POSTGRES_HOST_AUTH_METHOD=trust \
-e POSTGRES_INITDB_ARGS="--locale=C" \
-p 5432:5432 \
docker.io/library/postgres:${{ matrix.pg }}

- name: Cache Go dependencies
uses: ./.github/actions/cache-go-dependencies
with:
key-suffix: ${{ matrix.gotags }}
key-suffix: ${{ matrix.gotags }}-${{ matrix.shard.name }}

- name: Is Postgres ready
run: pg_isready -h 127.0.0.1
run: |
for _ in $(seq 1 60); do
docker exec postgres pg_isready -h 127.0.0.1 && exit 0
sleep 1
done
echo "Postgres failed to become ready"
docker logs postgres
exit 1

- name: Go Unit Tests
run: ${{ matrix.gotags }} make go-postgres-unit-tests
run: SKIP_DEPS=1 ${{ matrix.gotags }} make ${{ matrix.shard.make-target }}

- uses: codecov/codecov-action@v3
with:
Expand Down Expand Up @@ -178,35 +298,49 @@ jobs:

go-bench:
runs-on: ubuntu-latest
container:
image: quay.io/stackrox-io/apollo-ci:stackrox-test-0.5.3
volumes:
- /usr:/mnt/usr
- /opt:/mnt/opt
steps:
- name: Set Postgres version
run: |
echo "/usr/pgsql-15/bin" >> "${GITHUB_PATH}"

- name: Checkout
uses: actions/checkout@v6
with:
fetch-depth: 0

- uses: actions/setup-go@v6
with:
go-version-file: go.mod
cache: false

# setup-go exports GOTOOLCHAIN matching the go.mod value, which
# prevents auto-downloading newer toolchains. Override to auto so
# sub-modules with newer go directives (e.g. tools/proto) work.
- name: Override GOTOOLCHAIN
run: echo "GOTOOLCHAIN=auto" >> "$GITHUB_ENV"

- uses: ./.github/actions/job-preamble
with:
gcp-account: ${{ secrets.GCP_SERVICE_ACCOUNT_STACKROX_CI }}

# trust: allow passwordless TCP connections (replaces unix socket peer auth from CI container)
# locale=C: deterministic collation matching CI container's default
- name: Run Postgres
run: |
su postgres -c 'initdb -D /tmp/data'
su postgres -c 'pg_ctl -D /tmp/data start'
docker run --rm -d --name postgres \
-e POSTGRES_HOST_AUTH_METHOD=trust \
-e POSTGRES_INITDB_ARGS="--locale=C" \
-p 5432:5432 \
docker.io/library/postgres:15

- name: Cache Go dependencies
uses: ./.github/actions/cache-go-dependencies

- name: Is Postgres ready
run: pg_isready -h 127.0.0.1
run: |
for _ in $(seq 1 60); do
docker exec postgres pg_isready -h 127.0.0.1 && exit 0
sleep 1
done
echo "Postgres failed to become ready"
docker logs postgres
exit 1

- name: Go Bench Tests
run: make go-postgres-bench-tests
Expand Down Expand Up @@ -501,6 +635,8 @@ jobs:
runs-on: ubuntu-latest
needs:
- go
- go-integration
- go-operator-integration
- go-bench
- go-postgres
- local-roxctl-tests
Expand Down
16 changes: 14 additions & 2 deletions Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -342,9 +342,15 @@ config-controller-gen:
.PHONY: generated-srcs
generated-srcs: go-generated-srcs config-controller-gen

ifdef SKIP_DEPS
deps:
@echo "+ $@ (skipped via SKIP_DEPS)"
$(SILENT)touch deps
else
deps: $(shell find $(BASE_DIR) -name "go.sum")
@echo "+ $@"
$(SILENT)touch deps
endif

%/go.sum: %/go.mod
$(SILENT)cd $*
Expand Down Expand Up @@ -554,11 +560,17 @@ sensor-pipeline-benchmark: build-prep test-prep
LOGLEVEL="panic" go test -bench=. -run=^# -benchtime=30s -count=5 ./sensor/tests/pipeline | tee $(CURDIR)/test-output/pipeline.results.txt

.PHONY: go-postgres-unit-tests
go-postgres-unit-tests: build-prep test-prep
go-postgres-unit-tests: go-postgres-unit-tests-main go-postgres-unit-tests-migrator

.PHONY: go-postgres-unit-tests-main
go-postgres-unit-tests-main: build-prep test-prep
set -o pipefail ; \
CGO_ENABLED=1 GOEXPERIMENT=cgocheck2 MUTEX_WATCHDOG_TIMEOUT_SECS=30 GOTAGS=$(GOTAGS),test,sql_integration scripts/go-test.sh -timeout 15m -race -cover -coverprofile test-output/coverage.out -v \
$(shell git grep -rl "//go:build sql_integration" central pkg tools | sed -e 's@^@./@g' | xargs -n 1 dirname | sort | uniq | xargs go list -tags sql_integration | grep -v '^github.com/stackrox/rox/tests$$' | grep -Ev $(UNIT_TEST_IGNORE)) \
| tee $(GO_TEST_OUTPUT_PATH)

.PHONY: go-postgres-unit-tests-migrator
go-postgres-unit-tests-migrator: build-prep test-prep
@# The -p 1 passed to go test is required to ensure that tests of different packages are not run in parallel, so as to avoid conflicts when interacting with the DB.
set -o pipefail ; \
CGO_ENABLED=1 GOEXPERIMENT=cgocheck2 MUTEX_WATCHDOG_TIMEOUT_SECS=30 GOTAGS=$(GOTAGS),test,sql_integration scripts/go-test.sh -p 1 -race -cover -coverprofile test-output/migrator-coverage.out -v \
Expand Down Expand Up @@ -602,7 +614,7 @@ test: go-unit-tests ui-test shell-unit-tests
.PHONY: integration-unit-tests
integration-unit-tests: build-prep test-prep
set -o pipefail ; \
GOTAGS=$(GOTAGS),test,integration scripts/go-test.sh -count=1 -v \
GOTAGS=$(GOTAGS),test,integration scripts/go-test.sh -v \
$(shell go list ./... | grep "registries\|scanners\|notifiers") \
| tee $(GO_TEST_OUTPUT_PATH)

Expand Down
Loading
Loading