perf(ci): stabilize file mtimes for Go test cache by davdhacs · Pull Request #19395 · stackrox/stackrox

davdhacs · 2026-03-12T14:42:56Z

Description

Go's test cache validates inputs by (path, size, mtime) — not content. git checkout sets all mtimes to "now", so test results are never cached across CI runs.

Fix: set tracked file and directory mtimes to a fixed date after checkout:

git ls-files | xargs touch -t 200101010000
git ls-files | sed 's|/[^/]*$||' | sort -u | xargs touch -t 200101010000

Directory mtimes matter because tests using os.ReadDir/filepath.Walk (e.g. helm chart tests) check them too. git ls-files only lists files.

Combined with #19617 (stable test ldflags), this enables full test caching on current master baseline: go 33m -> 7.2m (4.6x), go-postgres 31m -> 3.6m (8.6x). See #19618 for combined demo.

Prior art

#15609 by @janisz applied this with git-restore-mtime. It was closed due to two blockers at that time:

GOTAGS matrix collision — release and non-release tests shared one cache, invalidating each other. Now fixed by fix(ci): GHA cache keys unique vs shared #19580 (key-suffix).
-coverprofile disabled test caching in Go <1.25. Now fixed in Go 1.25.

Risk

If a testdata file changes content but not size between commits, the test cache could return a stale result. This matches Go's own documented limitation.

User-facing documentation

CHANGELOG.md is updated OR update is not needed
documentation PR is created and is linked above OR is not needed

Testing and quality

the change is production ready: the change is GA, or otherwise the functionality is gated by a feature flag
CI results are inspected

Automated testing

modified existing tests

How I validated my change

CI green on all unit-test jobs
Combined branch (perf(ci): enable full Go test caching (fixed ldflags + stable mtimes) #19618) warm run verified: go 33m -> 7.2m (4.6x), go-postgres 31m -> 3.6m (8.6x)

Partially generated by AI.

openshift-ci · 2026-03-12T14:43:01Z

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

sourcery-ai

Hey - I've found 1 issue, and left some high level feedback:

Consider restricting the git ls-files filter (e.g., to *.go and known testdata paths) to avoid unnecessarily touching large/binary/irrelevant tracked files and potentially interfering with tools that rely on mtimes for non-code assets.
It may be safer to make the fixed timestamp explicitly UTC-aware (e.g., TZ=UTC touch -t 200101010000 or touch -d '2001-01-01T00:00:00Z') to avoid any subtle differences across runners with different default timezones.

Prompt for AI Agents

Please address the comments from this code review:

## Overall Comments
- Consider restricting the `git ls-files` filter (e.g., to `*.go` and known testdata paths) to avoid unnecessarily touching large/binary/irrelevant tracked files and potentially interfering with tools that rely on mtimes for non-code assets.
- It may be safer to make the fixed timestamp explicitly UTC-aware (e.g., `TZ=UTC touch -t 200101010000` or `touch -d '2001-01-01T00:00:00Z'`) to avoid any subtle differences across runners with different default timezones.

## Individual Comments

### Comment 1
<location path=".github/actions/cache-go-dependencies/action.yaml" line_range="68" />
<code_context>
+        # Risk: if a non-.go testdata file changes content but not size between
+        # commits, the test cache could return a stale result. This is extremely
+        # rare and Go's own test cache has the same inherent limitation.
+        git ls-files -z | xargs -0 touch -t 200101010000
+      shell: bash
</code_context>
<issue_to_address>
**issue (bug_risk):** The fixed timestamp may still differ between runners due to local timezone interpretation.

`touch -t 200101010000` uses the runner’s local timezone. If different runners use different timezones (e.g., self‑hosted vs GitHub‑hosted), the resulting mtimes/epoch values will differ and the Go test cache may still miss across runners. To keep mtimes stable, force UTC (e.g., `TZ=UTC git ls-files -z | xargs -0 touch -t 200101010000`) or use a UTC `-d '2001-01-01T00:00:00Z'` style argument where available.
</issue_to_address>

Sourcery is free for open source - if you like our reviews please consider sharing them ✨

_{Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.}

.github/actions/cache-go-dependencies/action.yaml

rhacs-bot · 2026-03-12T15:02:22Z

Images are ready for the commit at 2087aec.

To use with deploy scripts, first export MAIN_IMAGE_TAG=4.11.x-510-g2087aec9fd.

codecov · 2026-03-12T15:23:41Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 49.66%. Comparing base (92a755a) to head (2087aec).
⚠️ Report is 4 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master   #19395      +/-   ##
==========================================
- Coverage   49.67%   49.66%   -0.01%     
==========================================
  Files        2748     2748              
  Lines      207354   207354              
==========================================
- Hits       102996   102986      -10     
- Misses      96704    96712       +8     
- Partials     7654     7656       +2

Flag	Coverage Δ
go-unit-tests	`49.66% <ø> (-0.01%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

janisz · 2026-03-13T12:34:34Z

tests(CI): restore file mtime #15609 (comment)

Go's test cache computes a test binary ID that includes -X ldflags. Since -X MainVersion and -X GitShortSha change on every commit, the test binary ID changes too, causing 100% test cache misses (Phase 1). Fix: generate pkg/version/internal/zversion.go with an init() function instead of passing version data via -X ldflags. For tests, use only the base tag (e.g. "4.11.x") which is stable across commits. For builds, use the full git-describe version. This eliminates -X ldflags entirely from the link step, stabilizing the link ActionID. Result: test binary IDs are constant across commits (fixing Phase 1), and build cache hits improve from ~0% to ~99.9% (only 2/4403 packages recompile — version packages whose output is byte-identical anyway). Also adds -buildvcs=false to disable Go 1.18+'s unused VCS stamping, which otherwise adds git state to every binary. This is the companion to PR #19395 (mtime fix for Phase 2). Together they enable 4.5-52x test speedup. Addresses concerns from PR #15609: -coverprofile caching is fixed in Go 1.25 (we're on 1.25.5), and GOTAGS=release shares 93% of cache entries with default — separation is counterproductive. Partially generated by AI. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Go's test cache computes a test binary ID that includes -X ldflags. Since -X MainVersion and -X GitShortSha change on every commit, the test binary ID changes too, causing 100% test cache misses (Phase 1). Fix: generate pkg/version/internal/zversion.go with an init() function instead of passing version data via -X ldflags. For tests, use only the base tag (e.g. "4.11.x") which is stable across commits. For builds, use the full git-describe version. This eliminates -X ldflags entirely from the link step, stabilizing the link ActionID. Result: test binary IDs are constant across commits (fixing Phase 1), and build cache hits improve from ~0% to ~99.9% (only 2/4403 packages recompile — version packages whose output is byte-identical anyway). This is the companion to PR #19395 (mtime fix for Phase 2). Together they enable 4.5-52x test speedup. Addresses concerns from PR #15609: -coverprofile caching is fixed in Go 1.25 (we're on 1.25.5), and GOTAGS=release shares 93% of cache entries with default — separation is counterproductive. Partially generated by AI. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

davdhacs · 2026-03-24T22:51:41Z

@sourcery-ai review

sourcery-ai

Hey - I've left some high level feedback:

Consider constraining the git ls-files scope (e.g., excluding large generated/vendor directories if not needed for tests) to avoid unnecessary touch operations and minimize overhead on large repos.
If tests rely on untracked/generated files (e.g., testdata produced at build time), note that these files will still have non-deterministic mtimes and bypass the cache; you may want to either document this explicitly in the comment block or extend the approach to cover those paths if relevant.

Prompt for AI Agents

Please address the comments from this code review:

## Overall Comments
- Consider constraining the `git ls-files` scope (e.g., excluding large generated/vendor directories if not needed for tests) to avoid unnecessary `touch` operations and minimize overhead on large repos.
- If tests rely on untracked/generated files (e.g., testdata produced at build time), note that these files will still have non-deterministic mtimes and bypass the cache; you may want to either document this explicitly in the comment block or extend the approach to cover those paths if relevant.

Sourcery is free for open source - if you like our reviews please consider sharing them ✨

_{Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.}

Go's test cache computes a test binary ID that includes -X ldflags. Since -X MainVersion and -X GitShortSha change on every commit, the test binary ID changes too, causing 100% test cache misses (Phase 1). Fix: generate pkg/version/internal/zversion.go with an init() function instead of passing version data via -X ldflags. For tests, use only the base tag (e.g. "4.11.x") which is stable across commits. For builds, use the full git-describe version. This eliminates -X ldflags entirely from the link step, stabilizing the link ActionID. Result: test binary IDs are constant across commits (fixing Phase 1), and build cache hits improve from ~0% to ~99.9% (only 2/4403 packages recompile — version packages whose output is byte-identical anyway). This is the companion to PR #19395 (mtime fix for Phase 2). Together they enable 4.5-52x test speedup. Addresses concerns from PR #15609: -coverprofile caching is fixed in Go 1.25 (we're on 1.25.5), and GOTAGS=release shares 93% of cache entries with default — separation is counterproductive. Partially generated by AI. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Tests don't need real version info. Use fixed version strings (0.0.0-test) instead of git-describe values for test ldflags. This makes link ActionIDs stable across commits, enabling Go's test result cache to hit. Also adds -buildvcs=false to stop Go 1.18+'s VCS stamping which independently changes link ActionIDs on every commit. Builds keep the current XDef/status.sh mechanism unchanged. Verified locally: tests cache across commits with fixed ldflags. Combined with the mtime fix (#19395), this enables full warm test caching in CI (2.5-8.2x faster test jobs). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

davdhacs · 2026-03-25T17:52:15Z

Note: #19617 is a minimal companion fix that stabilizes test ldflags (Phase 1 of test caching). Combined with this PR's mtime fix (Phase 2), full test caching is enabled — see #19618 for the combined branch with expected 2.5-8x test speedup.

Tests don't need real version info. Use fixed version strings (0.0.0-test) instead of git-describe values for test ldflags. This makes link ActionIDs stable across commits, enabling Go's test result cache to hit. Also adds -buildvcs=false to stop Go 1.18+'s VCS stamping which independently changes link ActionIDs on every commit. Builds keep the current XDef/status.sh mechanism unchanged. Verified locally: tests cache across commits with fixed ldflags. Combined with the mtime fix (#19395), this enables full warm test caching in CI (2.5-8.2x faster test jobs). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

coderabbitai · 2026-03-30T17:13:40Z

📝 Walkthrough

Summary by CodeRabbit

Chores
- Improved build cache reliability to achieve better consistency and performance across development and test runs.

Walkthrough

A new composite-action step was added to stabilize Go test cache by setting deterministic file modification times (UTC timestamp 200101010000) for tracked repository files and directories, replacing nondeterministic timestamps from git checkout operations to improve test cache hits across runs.

Changes

Cohort / File(s)	Summary
Go Test Cache Stabilization `.github/actions/cache-go-dependencies/action.yaml`	Added a new workflow step that runs `touch -t 200101010000` on all tracked files and parent directories to normalize filesystem mtimes for Go's test cache, which uses path/size/mtime as cache input rather than file content.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~10 minutes

🚥 Pre-merge checks | ✅ 3

✅ Passed checks (3 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title accurately and concisely summarizes the main change: stabilizing file mtimes to improve Go test cache performance in CI.
Description check	✅ Passed	The PR description comprehensively covers all required template sections with technical depth, prior art context, risk assessment, and validation results.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests
Commit unit tests in branch davdhacs/test-cache-mtime

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

coderabbitai

🧹 Nitpick comments (1)

.github/actions/cache-go-dependencies/action.yaml (1)

108-112: Consider using null-delimited output with xargs -0 for robustness with special filenames.

While the repository currently has no files with spaces or special characters, using git ls-files -z | xargs -0 is a defensive best practice that prevents potential issues if such files are added in the future. Additionally, the first TZ=UTC on line 108 is unnecessary since git ls-files doesn't use the timezone variable.

♻️ Optional improvement for null-delimited safety

-    TZ=UTC git ls-files | xargs TZ=UTC touch -t 200101010000
+    git ls-files -z | TZ=UTC xargs -0 touch -t 200101010000
     # Also stabilize directory mtimes — tests using os.ReadDir or
     # filepath.Walk check directory mtimes too. git ls-files only
     # lists files, not directories.
-    git ls-files | sed 's|/[^/]*$||' | sort -u | xargs TZ=UTC touch -t 200101010000
+    git ls-files -z | sed -z 's|/[^/]*$||' | sort -zu | TZ=UTC xargs -0 touch -t 200101010000

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In @.github/actions/cache-go-dependencies/action.yaml around lines 108 - 112,
Replace the two pipelines that call git ls-files and xargs with null-delimited
variants and drop the unnecessary TZ=UTC before git ls-files; specifically, use
git ls-files -z piped into xargs -0 when touching file mtimes, and for
directories derive unique parent dirs from the null-delimited list (e.g. using
dirname/perl/awk on the -z stream) then sort uniquely and feed to xargs -0 touch
-t to set mtimes — update the lines that currently invoke "TZ=UTC git ls-files |
xargs TZ=UTC touch -t 200101010000" and "git ls-files | sed 's|/[^/]*$||' | sort
-u | xargs TZ=UTC touch -t 200101010000" to use the null-delimited, xargs -0
approach and remove the extraneous TZ=UTC before git ls-files.

🤖 Prompt for all review comments with AI agents

Verify each finding against the current code and only fix it if needed.

Nitpick comments:
In @.github/actions/cache-go-dependencies/action.yaml:
- Around line 108-112: Replace the two pipelines that call git ls-files and
xargs with null-delimited variants and drop the unnecessary TZ=UTC before git
ls-files; specifically, use git ls-files -z piped into xargs -0 when touching
file mtimes, and for directories derive unique parent dirs from the
null-delimited list (e.g. using dirname/perl/awk on the -z stream) then sort
uniquely and feed to xargs -0 touch -t to set mtimes — update the lines that
currently invoke "TZ=UTC git ls-files | xargs TZ=UTC touch -t 200101010000" and
"git ls-files | sed 's|/[^/]*$||' | sort -u | xargs TZ=UTC touch -t
200101010000" to use the null-delimited, xargs -0 approach and remove the
extraneous TZ=UTC before git ls-files.

ℹ️ Review info

⚙️ Run configuration

Configuration used: Central YAML (base), Organization UI (inherited)

Review profile: CHILL

Plan: Pro

Run ID: 7a0df54c-8fc3-4710-ba58-e36db977dfac

📥 Commits

Reviewing files that changed from the base of the PR and between 583e8d7 and ba5f8d2.

📒 Files selected for processing (1)

.github/actions/cache-go-dependencies/action.yaml

Tests don't need real version info. Use fixed version strings (0.0.0-test) instead of git-describe values for test ldflags. This makes link ActionIDs stable across commits, enabling Go's test result cache to hit. Also adds -buildvcs=false to stop Go 1.18+'s VCS stamping which independently changes link ActionIDs on every commit. Builds keep the current XDef/status.sh mechanism unchanged. Verified locally: tests cache across commits with fixed ldflags. Combined with the mtime fix (#19395), this enables full warm test caching in CI (2.5-8.2x faster test jobs). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Go's test cache validates inputs by (path, size, mtime) — not content. git checkout sets all file mtimes to the current time, which differs between CI runs, causing test results to never be cached. Fix: set all tracked file mtimes to a fixed date (2001-01-01) after checkout and cache restore. This makes the test cache hit across runs when file content hasn't changed. Measured impact (from PR #19201): - go unit tests: 35m → 8m (4.5x) - go-postgres: 31m → 36s (52x) - sensor-integration: 26m → 2.5m (11x) - Test cache hit rate: 97.6% (681/698 packages) Partially generated by AI. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

git ls-files only outputs files, not directories. Tests that read directories at runtime (os.ReadDir, filepath.Walk) check directory mtimes for cache validation. Without stable dir mtimes, these tests always miss the cache. The helm chart tests (pkg/helm/charts/tests/*) use testdata directories and were the largest uncached tests at 712s combined. This fix caches them, dropping the Go Unit Tests step from 486s to 85s. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Ensures consistent epoch values regardless of runner timezone. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

…-mtime

TZ=UTC on git ls-files was causing shell issues. All GHA runners are UTC by default, so TZ=UTC on touch is unnecessary. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

.github/actions/cache-go-dependencies/action.yaml

…-mtime Resolve conflict: keep mtime stabilization step, drop removed 'Download Go modules' step (removed by #19688 GOMODCACHE PR). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

davdhacs added area/ci ai-assisted labels Mar 12, 2026

openshift-ci bot added the do-not-merge/work-in-progress label Mar 12, 2026

github-actions bot added the ai-review label Mar 12, 2026

sourcery-ai bot reviewed Mar 12, 2026

View reviewed changes

.github/actions/cache-go-dependencies/action.yaml Outdated Show resolved Hide resolved

davdhacs force-pushed the davdhacs/test-cache-mtime branch from 05f5ad5 to 21752e5 Compare March 12, 2026 16:53

davdhacs changed the base branch from davdhacs/zversion-codegen to master March 12, 2026 16:53

davdhacs force-pushed the davdhacs/test-cache-mtime branch from 21752e5 to f13f6a2 Compare March 12, 2026 17:41

github-actions bot added the area/operator label Mar 12, 2026

This was referenced Mar 12, 2026

perf(ci): use stable test ldflags for Go test cache #19394

Closed

perf(ci): test cache — combined stable ldflags + mtime fix #19404

Closed

davdhacs force-pushed the davdhacs/test-cache-mtime branch from f13f6a2 to 38d32df Compare March 24, 2026 20:29

davdhacs mentioned this pull request Mar 24, 2026

perf(ci): version refactor (remove Xref/status.sh) #19424

Closed

6 tasks

sourcery-ai bot reviewed Mar 24, 2026

View reviewed changes

davdhacs marked this pull request as ready for review March 25, 2026 04:07

davdhacs requested a review from a team as a code owner March 25, 2026 04:07

openshift-ci bot removed the do-not-merge/work-in-progress label Mar 25, 2026

davdhacs requested a review from janisz March 25, 2026 04:07

This was referenced Mar 25, 2026

perf(ci): use stable version ldflags for test caching #19617

Merged

perf(ci): enable full Go test caching (fixed ldflags + stable mtimes) #19618

Closed

davdhacs force-pushed the davdhacs/test-cache-mtime branch 3 times, most recently from 8bae563 to db06047 Compare March 26, 2026 04:23

davdhacs force-pushed the davdhacs/test-cache-mtime branch from db06047 to 2ba77eb Compare March 28, 2026 05:00

davdhacs mentioned this pull request Mar 29, 2026

perf(ci): remove container from unit-tests jobs #19678

Draft

5 tasks

github-actions bot added the coderabbit-review label Mar 30, 2026

coderabbitai bot reviewed Mar 30, 2026

View reviewed changes

davdhacs and others added 3 commits March 31, 2026 07:06

fix: use TZ=UTC for mtime stabilization

b313cbf

Ensures consistent epoch values regardless of runner timezone. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

davdhacs force-pushed the davdhacs/test-cache-mtime branch from ba5f8d2 to b313cbf Compare March 31, 2026 13:06

davdhacs and others added 2 commits March 31, 2026 09:17

Merge remote-tracking branch 'origin/master' into davdhacs/test-cache…

83baecf

…-mtime

fix: remove TZ=UTC from mtime stabilization (all runners are UTC)

a04654b

TZ=UTC on git ls-files was causing shell issues. All GHA runners are UTC by default, so TZ=UTC on touch is unnecessary. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

davdhacs commented Mar 31, 2026

View reviewed changes

.github/actions/cache-go-dependencies/action.yaml Show resolved Hide resolved

janisz reviewed Mar 31, 2026

View reviewed changes

.github/actions/cache-go-dependencies/action.yaml Show resolved Hide resolved

janisz approved these changes Mar 31, 2026

View reviewed changes

davdhacs enabled auto-merge (squash) March 31, 2026 17:51

Merge remote-tracking branch 'origin/master' into davdhacs/test-cache…

2087aec

…-mtime Resolve conflict: keep mtime stabilization step, drop removed 'Download Go modules' step (removed by #19688 GOMODCACHE PR). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

davdhacs merged commit 7081e0a into master Mar 31, 2026
105 checks passed

davdhacs deleted the davdhacs/test-cache-mtime branch March 31, 2026 20:34

Conversation

davdhacs commented Mar 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Prior art

Risk

User-facing documentation

Testing and quality

Automated testing

How I validated my change

Uh oh!

openshift-ci bot commented Mar 12, 2026

Uh oh!

sourcery-ai bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

rhacs-bot commented Mar 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov bot commented Mar 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

janisz commented Mar 13, 2026

Uh oh!

davdhacs commented Mar 24, 2026

Uh oh!

sourcery-ai bot left a comment

Choose a reason for hiding this comment

Uh oh!

davdhacs commented Mar 25, 2026

Uh oh!

coderabbitai bot commented Mar 30, 2026

Summary by CodeRabbit

Walkthrough

Changes

Estimated code review effort

Uh oh!

coderabbitai bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

davdhacs commented Mar 12, 2026 •

edited

Loading

rhacs-bot commented Mar 12, 2026 •

edited

Loading

codecov bot commented Mar 12, 2026 •

edited

Loading