ci(experiment): CPU/cache diagnostics matrix#21044
Draft
davdhacs wants to merge 12 commits into
Draft
Conversation
Investigate whether Intel Xeon vs AMD EPYC runners produce different Go compile actionIDs, causing test cache misses across runner types. Changes to the go job in unit-tests.yaml: - Scale to 10 matrix copies (GOTAGS="" only) to sample runner hardware - Add post-test diagnostics step logging: CPU model, GOCACHE size, test cache hit rate, and compile actionIDs for canary packages - Strip non-essential steps (codecov, junit2jira, operator/integration) to keep experiment focused and fast Results will appear in each job's Step Summary for easy comparison. Prompt: set up a branch for collecting data on build time, cpuinfo, compilation cache, and used cached test results with a matrix dimension to sample CPU types Partially generated by AI. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
|
Skipping CI for Draft Pull Request. |
Contributor
|
Important Review skippedAuto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Enterprise Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Contributor
🚀 Build Images ReadyImages are ready for commit 8bf0f81. To use with deploy scripts: export MAIN_IMAGE_TAG=4.12.x-134-g8bf0f814cb |
Runs 3 test packages (pkg/set, central/cluster/util, central/notifiers/slack) with gocachetest=1 on copy 1 before the main test run. The GODEBUG output goes to Step Summary showing Phase 1 vs Phase 2 miss reason. Also removes -trimpath from the main test (back to matching CI behavior) since we need to diagnose the current 78% hit rate, not test a fix.
The previous fix only stabilized BUILD_TAG and SHORTCOMMIT, leaving CollectorVersion, FactVersion, and ScannerVersion changing per commit. These unstable ldflags propagate through go-test.sh → status.sh → -X flags → link actionID → test binary ID, invalidating the test cache for every package that transitively depends on pkg/version/internal. This was the root cause of the 78% cache hit rate on stale branches (vs 96% on fresh branches where the versions happen to match the cache). Fix: add env var overrides in status.sh for STABLE_COLLECTOR_VERSION, STABLE_FACT_VERSION, STABLE_SCANNER_VERSION, set to 0.0.0 in the unit-tests workflow. Normal builds (without the env vars) are unaffected. Expected result: cache hit rate should match the 96% baseline regardless of how far behind master the branch is. Partially generated by AI.
davdhacs
force-pushed
the
davdhacs/cpu-cache-experiment
branch
from
1448025 to
7eea672
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Experiment to investigate whether Intel Xeon vs AMD EPYC GHA runners produce
different Go compile actionIDs, causing test cache misses when the GOCACHE is
shared across runner types. We observed identical code, identical cache keys,
but 11% vs 94% test cache hit rate depending on which CPU the runner had.
Scales the
gojob to 10 matrix copies (GOTAGS="" only) to sample runnerhardware diversity. Each copy logs CPU model, GOCACHE size, test cache hit
rate, and compile actionIDs for canary packages to the Step Summary.
Not intended to merge. Experiment only.
User-facing documentation
Testing and quality
Automated testing
How I validated my change
Experiment PR — validation is the CI run output itself. Will compare Step
Summaries across the 10 copies to correlate CPU model with cache hit rate
and compile actionID differences.