Skip to content

fix(ci): use IfNotPresent pull policy for prefetched images on non-GKE #20993

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
davdhacs wants to merge 2 commits into
base: master
Choose a base branch
from
+21 −16
Open

fix(ci): use IfNotPresent pull policy for prefetched images on non-GKE #20993

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
e580b52
fix(ci): use IfNotPresent instead of Never pull policy on non-GKE clu…
davdhacs Jun 5, 2026
a75068d
Merge remote-tracking branch 'origin/master' into worktree-fix-ocp-pr…
davdhacs Jun 5, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
37 changes: 21 additions & 16 deletions scripts/ci/lib.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -612,31 +612,19 @@ _image_prefetcher_prebuilt_start() {
case "$CI_JOB_NAME" in
*qa-e2e-tests)
image_prefetcher_start_set qa-e2e
# Override the default image pull policy for containers with quay.io
# images to rely on prefetched images. This helps ensure that the static
# prefect list stays up to date with additions.
ci_export "IMAGE_PULL_POLICY_FOR_QUAY_IO" "Never"
_set_quay_pull_policy
;;
*nongroovy-e2e-tests)
image_prefetcher_start_set qa-nongroovy-e2e
# Override the default image pull policy for containers with quay.io
# images to rely on prefetched images. This helps ensure that the static
# prefect list stays up to date with additions.
ci_export "IMAGE_PULL_POLICY_FOR_QUAY_IO" "Never"
_set_quay_pull_policy
;;
*nongroovy-compatibility-tests)
image_prefetcher_start_set nongroovy-compatibility
# Override the default image pull policy for containers with quay.io
# images to rely on prefetched images. This helps ensure that the static
# prefect list stays up to date with additions.
ci_export "IMAGE_PULL_POLICY_FOR_QUAY_IO" "Never"
_set_quay_pull_policy
;;
*compatibility-tests)
image_prefetcher_start_set compatibility
# Override the default image pull policy for containers with quay.io
# images to rely on prefetched images. This helps ensure that the static
# prefect list stays up to date with additions.
ci_export "IMAGE_PULL_POLICY_FOR_QUAY_IO" "Never"
_set_quay_pull_policy
;;
*-operator-e2e-tests)
image_prefetcher_start_set operator-e2e
Expand All @@ -648,6 +636,23 @@ _image_prefetcher_prebuilt_start() {
esac
}

# Override imagePullPolicy for quay.io images to prefer prefetched images.
# On providers with kubelet credential plugin integration (GKE), use Never
# to enforce that the prefetch list stays complete — any missing image fails
# loudly. On providers without credential plugin integration (OCP, EKS, AKS),
# use IfNotPresent instead, because Kubernetes 1.35+ enforces credential
# provenance checks with Never policy: the kubelet rejects pre-pulled images
# unless the pod's credentials match a recorded pull (KEP-2535). The prefetcher
# pulls images using its own credentials, so the test pod's credentials don't
# match, causing ErrImageNeverPull even though the image is on the node.
Comment on lines +640 to +647

@porridge porridge Jun 11, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
# On providers with kubelet credential plugin integration (GKE), use Never
# to enforce that the prefetch list stays complete — any missing image fails
# loudly. On providers without credential plugin integration (OCP, EKS, AKS),
# use IfNotPresent instead, because Kubernetes 1.35+ enforces credential
# provenance checks with Never policy: the kubelet rejects pre-pulled images
# unless the pod's credentials match a recorded pull (KEP-2535). The prefetcher
# pulls images using its own credentials, so the test pod's credentials don't
# match, causing ErrImageNeverPull even though the image is on the node.
# Unfortunately https://github.com/kubernetes/kubernetes/issues/138175 broke
# this for a handful of images that are also pulled from another registry
# (but have the same content digest).
# On GKE, we worked this around with kubelet credential plugin integration in the image
# prefetcher, so in this case we can use `Never` to enforce that the prefetch list stays
# complete — any image missing from prefetch list fails loudly.
# On providers (OCP, EKS, AKS) that do not have credential plugin integration (yet),
# use IfNotPresent instead.
# TODO(ROX-35031): set this unconditionally to Never when a proper fix is available in all supported OCP versions

_set_quay_pull_policy() {
local policy="Never"
if [[ "${KUBERNETES_PROVIDER}" != "gke" ]]; then
policy="IfNotPresent"
fi
ci_export "IMAGE_PULL_POLICY_FOR_QUAY_IO" "$policy"
}

_image_prefetcher_system_start() {
# NOTE: when changing this function, make corresponding changes to
# _image_prefetcher_system_await
Expand Down
Loading