fix(ci): use IfNotPresent pull policy for prefetched images on non-GKE#20993
fix(ci): use IfNotPresent pull policy for prefetched images on non-GKE#20993davdhacs wants to merge 2 commits into
Conversation
…sters Kubernetes 1.35 (OCP 4.22) enables the KubeletEnsureSecretPulledImages feature gate by default (KEP-2535), which enforces credential provenance checks for pre-pulled images. With imagePullPolicy: Never, the kubelet now rejects images unless the pod's credentials match a recorded pull. The image-prefetcher pulls images using CRI with its own credentials, so test pods' credentials don't match — causing ErrImageNeverPull even though the image is present on the node. This broke all OCP 4.22 compat test runs (6+ consecutive failures on the quay-auto test). On GKE, this was solved by integrating the kubelet credential plugin into the prefetcher (image-prefetcher PR #184). On OCP/EKS/AKS where no credential plugin integration exists, use IfNotPresent instead of Never. The prefetch list hygiene goal (catching missing images) is still enforced on GKE which retains the Never policy. AI-assisted. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
|
Skipping CI for Draft Pull Request. |
|
/test gke-qa-e2e-tests |
|
/test ocp-4-22-nongroovy-e2e-tests |
|
/test ocp-4-12-nongroovy-e2e-tests |
|
Important Review skippedAuto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Enterprise Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
/test ocp-4-22-qa-e2e-tests |
🚀 Build Images ReadyImages are ready for commit a75068d. To use with deploy scripts: export MAIN_IMAGE_TAG=4.12.x-102-ga75068dbf9 |
|
For my understanding, does this mean we could also skip the kubelet credential plugin integration on GKE if we replace "Never" with "IfNotPresent"? |
Yes, I think so. But then we would not get a failure if someone adds an image to a test but does not add it to the image-prefetcher. -- I think @porridge had noted that part of the reason to use "Never" is to verify that the prefetch list covers all images used in tests. If we keep it as Never on at least one flavor, then we test that all images are prefetched still but avoid the registry credentials issue on OCP. I think that by setting |
|
re: image found but failure if |
@porridge Could we apply the IfNotPresent as a workaround for now until the image credential provider setup is merged? |
3 participants
Description
Kubernetes 1.35 (OCP 4.22) enables the
KubeletEnsureSecretPulledImagesfeature gate by default (KEP-2535). This changes the semantics ofimagePullPolicy: Never: the kubelet now rejects pre-pulled images unless the pod's credentials match a recorded pull.The image-prefetcher pulls images via CRI using its own credentials. On GKE, this is handled by kubelet credential plugin integration (image-prefetcher PR #184, stackrox PR #19287). On OCP/EKS/AKS, no credential plugin integration exists, so test pods get
ErrImageNeverPulleven though the image is present on the node.This caused 6+ consecutive failures of the
ImageScanningTest > Image metadata from registry test - quay-autotest on the OCP 4.22 periodic compat job.The fix: use
IfNotPresentinstead ofNeveron non-GKE clusters. The image will still come from the prefetch cache when present, but won't fail the credential provenance check. The prefetch list hygiene goal (catching missing images) is still enforced on GKE which retainsNever.User-facing documentation
Testing and quality
Automated testing
How I validated my change
Requesting targeted CI runs:
gke-qa-e2e-tests— verify GKE still usesNeverand passes (no regression)ocp-4-22-nongroovy-e2e-tests— verify OCP 4.22 (K8s 1.35) now usesIfNotPresentand passesocp-4-12-nongroovy-e2e-tests— verify older OCP still works withIfNotPresentThe periodic
ocp-4.22-lpMainline-lp-ocp-compat-cr--acs--tests-awsjob (which runs the groovyImageScanningTestincludingquay-auto) will validate on master after merge.