Skip to content

fix(ci): work around preloaded image use problem in recent k8s#19287

Merged
porridge merged 9 commits into
masterfrom
porridge/prefetch-digests
Apr 21, 2026
Merged

fix(ci): work around preloaded image use problem in recent k8s#19287
porridge merged 9 commits into
masterfrom
porridge/prefetch-digests

Conversation

@porridge

@porridge porridge commented Mar 4, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Description

This is a workaround for a kubernetes issue with image pull authorization. See:

This change:

  • increases disk size on GKE to make sure we're nowhere near the threshold for triggering GC
  • bumps image prefetcher to version that includes feat: initial image cred plugin integration image-prefetcher#184
  • activates kubelet-image-credential-integration when running on GKE
  • adds the problematic GCR-hosted images to prefetch list
  • skips prefetching GCR hosted images when running on OpenShift

TODO:

User-facing documentation

Testing and quality

  • the change is production ready: the change is GA, or otherwise the functionality is gated by a feature flag
  • CI results are inspected

Automated testing

  • modified existing tests

How I validated my change

CI is enough.

@openshift-ci

openshift-ci Bot commented Mar 4, 2026

Copy link
Copy Markdown

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

@porridge

porridge commented Mar 4, 2026

Copy link
Copy Markdown
Contributor Author

/test all

@porridge porridge mentioned this pull request Mar 4, 2026
Closed
@porridge

porridge commented Mar 4, 2026

Copy link
Copy Markdown
Contributor Author

/test gke-latest-qa-e2e-tests

@rhacs-bot

rhacs-bot commented Mar 4, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Images are ready for the commit at b00c781.

To use with deploy scripts, first export MAIN_IMAGE_TAG=4.11.x-536-gb00c7816cf.

@codecov

codecov Bot commented Mar 4, 2026 •
edited
Loading

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 49.68%. Comparing base (487e4d0) to head (c885cd2).

Additional details and impacted files 
@@           Coverage Diff           @@
##           master   #19287   +/-   ##
=======================================
  Coverage   49.68%   49.68%           
=======================================
  Files        2766     2766           
  Lines      209299   209299           
=======================================
  Hits       103994   103994           
  Misses      97613    97613           
  Partials     7692     7692
Flag Coverage Δ
go-unit-tests 49.68% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@porridge

porridge commented Mar 4, 2026

Copy link
Copy Markdown
Contributor Author

/test gke-latest-qa-e2e-tests

@porridge porridge force-pushed the porridge/prefetch-digests branch from 0f22fab to f19277c Compare March 26, 2026 08:43
@porridge

porridge commented Mar 26, 2026

Copy link
Copy Markdown
Contributor Author

/test gke-latest-qa-e2e-tests

6 similar comments
@porridge

porridge commented Mar 26, 2026

Copy link
Copy Markdown
Contributor Author

/test gke-latest-qa-e2e-tests

@porridge

porridge commented Mar 26, 2026

Copy link
Copy Markdown
Contributor Author

/test gke-latest-qa-e2e-tests

@porridge

porridge commented Mar 30, 2026

Copy link
Copy Markdown
Contributor Author

/test gke-latest-qa-e2e-tests

@porridge

porridge commented Mar 30, 2026

Copy link
Copy Markdown
Contributor Author

/test gke-latest-qa-e2e-tests

@porridge

porridge commented Mar 30, 2026

Copy link
Copy Markdown
Contributor Author

/test gke-latest-qa-e2e-tests

@porridge

porridge commented Mar 30, 2026

Copy link
Copy Markdown
Contributor Author

/test gke-latest-qa-e2e-tests

@porridge porridge force-pushed the porridge/prefetch-digests branch from 967f519 to bfd3b06 Compare March 31, 2026 11:10
@porridge

porridge commented Mar 31, 2026

Copy link
Copy Markdown
Contributor Author

/test gke-latest-qa-e2e-tests

1 similar comment
@porridge

porridge commented Apr 1, 2026

Copy link
Copy Markdown
Contributor Author

/test gke-latest-qa-e2e-tests

@porridge

porridge commented Apr 1, 2026

Copy link
Copy Markdown
Contributor Author

/test ?

@porridge

porridge commented Apr 1, 2026

Copy link
Copy Markdown
Contributor Author

/test gke-latest-nongroovy-e2e-tests
/test gke-latest-operator-e2e-tests
/test gke-latest-ui-e2e-tests

@porridge porridge force-pushed the porridge/prefetch-digests branch from b824d92 to b00c781 Compare April 1, 2026 11:35
@porridge

porridge commented Apr 1, 2026

Copy link
Copy Markdown
Contributor Author

/test gke-latest-nongroovy-e2e-tests
/test gke-latest-operator-e2e-tests
/test gke-latest-ui-e2e-tests

@porridge porridge force-pushed the porridge/prefetch-digests branch from b00c781 to 15448d3 Compare April 14, 2026 11:35
@porridge

porridge commented Apr 14, 2026

Copy link
Copy Markdown
Contributor Author

/test gke-latest-qa-e2e-tests
/test gke-qa-e2e-tests

@porridge

porridge commented Apr 14, 2026

Copy link
Copy Markdown
Contributor Author

/test ?

@porridge

porridge commented Apr 14, 2026

Copy link
Copy Markdown
Contributor Author

/test ocp-4-12-qa-e2e-tests

@github-actions

github-actions Bot commented Apr 14, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

🚀 Build Images Ready

Images are ready for commit a62ce25. To use with deploy scripts:

export MAIN_IMAGE_TAG=4.11.x-720-ga62ce25e52

mclasmeier
mclasmeier reviewed Apr 16, 2026
View reviewed changes
Comment thread scripts/ci/lib.sh Outdated
@porridge

porridge commented Apr 16, 2026

Copy link
Copy Markdown
Contributor Author

/test ?

@porridge

porridge commented Apr 16, 2026

Copy link
Copy Markdown
Contributor Author

/test aks-qa-e2e-tests
/test aro-qa-e2e-tests
/test eks-qa-e2e-tests
/test rosa-fips-qa-e2e-tests
/test rosa-hcp-qa-e2e-tests
/test rosa-qa-e2e-tests

@rhacs-bot

rhacs-bot commented Apr 16, 2026

Copy link
Copy Markdown
Contributor

/retest

@porridge

porridge commented Apr 20, 2026

Copy link
Copy Markdown
Contributor Author

/test gke-latest-nongroovy-e2e-tests
/test gke-latest-operator-e2e-tests
/test gke-latest-ui-e2e-tests

/test aks-qa-e2e-tests
/test aro-qa-e2e-tests
/test eks-qa-e2e-tests
/test rosa-fips-qa-e2e-tests
/test rosa-hcp-qa-e2e-tests
/test rosa-qa-e2e-tests

@porridge porridge closed this Apr 20, 2026
@porridge porridge reopened this Apr 20, 2026
@porridge

porridge commented Apr 20, 2026

Copy link
Copy Markdown
Contributor Author

/test gke-latest-nongroovy-e2e-tests
/test gke-latest-operator-e2e-tests
/test gke-latest-ui-e2e-tests

/test aks-qa-e2e-tests
/test aro-qa-e2e-tests
/test eks-qa-e2e-tests
/test rosa-fips-qa-e2e-tests
/test rosa-hcp-qa-e2e-tests
/test rosa-qa-e2e-tests

porridge and others added 9 commits April 20, 2026 10:43
@porridge
chore(ci): add a digest-qualified pull spec to nginx-1.12
b55ac90 
# Conflicts:
#	qa-tests-backend/scripts/images-to-prefetch.txt
@porridge
chore(ci): bump disk too
8a2d7ca
@porridge
quick stab at 2 gcr images
a38d5b6
@porridge
enable use-kubelet-image-credential-integration
df35e9e
@porridge
Filter out gcr.io images on non-GKE clusters
17022da
@porridge
bump
fc55952
@porridge
revert instance size change
be1dfbf
@porridge
avoid symlink attacks
1449bac
@porridge @claude
Add KUBERNETES_PROVIDER variable to distinguish cloud providers
c885cd2 
In CI job scripts, add KUBERNETES_PROVIDER env var set based on filename:
- gke_*.py → KUBERNETES_PROVIDER=gke
- eks_*.py → KUBERNETES_PROVIDER=eks
- aks_*.py → KUBERNETES_PROVIDER=aks
- ocp_*.py, rosa_*.py, osd_*.py, aro_*.py, powervs_*.py, ibmcloudz_*.py → KUBERNETES_PROVIDER=ocp

In scripts/ci/lib.sh:
- Use KUBERNETES_PROVIDER instead of ORCHESTRATOR_FLAVOR to control
  kubelet_image_creds (set to GKE only on GKE clusters)
- Filter out gcr.io images only on non-GKE clusters

This allows proper credential handling and image filtering specific to the
actual cloud provider rather than just the orchestrator flavor.

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
@porridge porridge force-pushed the porridge/prefetch-digests branch from 683681e to c885cd2 Compare April 20, 2026 08:43
@porridge

porridge commented Apr 20, 2026

Copy link
Copy Markdown
Contributor Author

/test gke-latest-nongroovy-e2e-tests
/test gke-latest-operator-e2e-tests
/test gke-latest-ui-e2e-tests

/test aks-qa-e2e-tests
/test aro-qa-e2e-tests
/test eks-qa-e2e-tests
/test rosa-fips-qa-e2e-tests
/test rosa-hcp-qa-e2e-tests
/test rosa-qa-e2e-tests

@tommartensen

tommartensen commented Apr 20, 2026

Copy link
Copy Markdown
Contributor

/test aro-qa-e2e-tests

@porridge

porridge commented Apr 20, 2026

Copy link
Copy Markdown
Contributor Author

EKS failure seems unrelated now: https://redhat.atlassian.net/browse/ROX-34160

@porridge

porridge commented Apr 20, 2026

Copy link
Copy Markdown
Contributor Author

the ocp-4-21-qa failures also seem like long-standing flakes

@openshift-ci

openshift-ci Bot commented Apr 20, 2026

Copy link
Copy Markdown

@porridge: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/prow/ocp-4-21-qa-e2e-tests c885cd2 link false /test ocp-4-21-qa-e2e-tests
ci/prow/eks-qa-e2e-tests c885cd2 link false /test eks-qa-e2e-tests
ci/prow/aro-qa-e2e-tests c885cd2 link false /test aro-qa-e2e-tests

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

@davdhacs

davdhacs commented Apr 20, 2026

Copy link
Copy Markdown
Contributor

+1

@porridge porridge requested a review from mclasmeier April 21, 2026 07:32
mclasmeier
mclasmeier approved these changes Apr 21, 2026
View reviewed changes

@mclasmeier mclasmeier left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a couple of questions.

Comment thread scripts/ci/jobs/aro_qa_e2e_tests.py
Comment thread scripts/ci/jobs/rosa_qa_e2e_tests.py
Comment thread scripts/ci/lib.sh
@porridge porridge merged commit a62ce25 into master Apr 21, 2026
110 of 113 checks passed
@porridge porridge deleted the porridge/prefetch-digests branch April 21, 2026 09:36
@porridge porridge mentioned this pull request May 6, 2026
Merged
@davdhacs davdhacs mentioned this pull request Jun 5, 2026
Open
5 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@coderabbitai coderabbitai[bot] coderabbitai[bot] left review comments

@vladbologa vladbologa vladbologa approved these changes

@davdhacs davdhacs davdhacs approved these changes

@mclasmeier mclasmeier mclasmeier approved these changes

@janisz janisz Awaiting requested review from janisz janisz is a code owner

Assignees

No one assigned

Labels

auto-retest PRs with this label will be automatically retested if prow checks fails

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@porridge @rhacs-bot @tommartensen @davdhacs @vladbologa @mclasmeier