Repository navigation
node-label: retry "Set label to node" while the Node object registers - #13621
romaincabassot wants to merge 1 commit into
Conversation
kubernetes-prow
Bot
added
release-note
|
Welcome @romaincabassot! |
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: romaincabassot The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
kubernetes-prow
Bot
added
the
needs-ok-to-test
|
Hi @romaincabassot. Thanks for your PR. I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with Regular contributors should join the org to skip this step. Once the patch is verified, the new status will be reflected by the I understand the commands that are listed here. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
One or more co-authors of this pull request were not found. You must specify co-authors in commit message trailer via: Supported
Alternatively, if the co-author should not be included, remove the Please update your commit message(s) by doing |
kubernetes-prow
Bot
added
the
cncf-cla: no
The Node object is created asynchronously by the kubelet after `kubeadm join` returns, so on a fresh install or scale-up the `kubectl label` call can run before the node is registered and fail with `Error from server (NotFound): nodes "<name>" not found`. Retry the task (10 times, 6s apart, same budget as the apiserver healthz wait earlier in the role), and only when the failure is a NotFound; any other error still fails immediately. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
7f4523e to
9d1c71b
Compare
|
/easycla |
What type of PR is this?
/kind bug
What this PR does / why we need it:
The
kubernetes/node-labelrole labels nodes withkubectl label node ...right after thekubernetes/kubeadmrole.kubeadm joinreturns once the kubelet's bootstrap CSR is signed, but the kubelet then has to switch to its new client certificate and register the Node object, which happens asynchronously. On a fresh install / scale-up (and on slower VMs or a busy apiserver) the label task can therefore run before the Node exists and fail the whole play (any_errors_fatal):Timeline from a real failing run (kubespray v2.30.0, 5-node cluster, Kubernetes 1.34):
kubeadm joinon node4 returned rc=0 at 15:58:28.2 ("This node has joined the cluster"), andkubectl label node node4hitNotFoundat 15:58:32.7, about 4 seconds later. The other nodes were labelled fine in the same step, which is why the failure is intermittent.The task has no
retries/until(the role only waits for the apiserver/healthz, not for the Node to exist). This PR adds a retry (10 x 6s, the same budget as the healthz wait), and only when the error isNotFound; any other failure (e.g. an invalid label) still fails immediately.Which issue(s) this PR is related to:
Related to #12779 (same symptom,
nodes "..." not foundinSet label to node; that report attributes it to CNI initialisation onscale.yml, this PR addresses the generic "Node not registered yet" window). I did not use "Fixes" since the root cause in that issue may differ.Special notes for your reviewer:
Only the
NotFoundcase is retried so real configuration errors are not delayed. I did not run the full CI matrix locally; the change was validated by parsing the task YAML and by analysing the failing run's timestamps.Does this PR introduce a user-facing change?
AI usage disclosure:
YES. The root-cause analysis and the patch were prepared with Claude Code (Claude Sonnet 5.5) and reviewed and submitted by the author, who is responsible for the change.