-
Notifications
You must be signed in to change notification settings - Fork 42k
kubelet: cloud-provider external addresses #121028
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
Kubelet, if using cloud provider external, initializes temporary the node addresses using the non-cloud provider logic, until the cloud provider overrides it. This behavior has undesired consequences if the cloud-provider addresses are different than the original ones, specially for hostNetwork pods, that inherit these addresses from the Node. Since some cloud-providers depend on this behavior, in order to keep backward compatibility, assume that the specifying addresses via the node-ip flags means that the intent is to keep the existing behavior to temporary initialize the addresses. If the node-ips are the unspecified addresses or are not set, then wait for the external cloud provider to set the node addresses. Change-Id: I3a3895f9b830769f9658e6a03f058c914c438a09 Signed-off-by: Antonio Ojea <aojea@google.com>
|
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the The DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
|
/assign @andrewsykim @danwinship @thockin I think that this approach solves the backwards compatiblity problem and the bug #120720 |
|
/priority important-soon |
|
/cc @SergeyKanzhelev |
|
/cc @tzneal @lubronzhan |
**What problem does this PR solve?**: Adds AWS CCM support for Kubernetes v1.29. While testing I also found an upstream change kubernetes/kubernetes#121028 and to fix that added `hostNetwork: true` to the CCM pods. Upstream CAPA templates and the Nutanix CCM already use `hostNetwork: true`, so it made sense to go with that approach here too. **Which issue(s) this PR fixes**: Fixes # **How Has This Been Tested?**: <!-- Please describe the tests that you ran to verify your changes. Provide output from the tests and any manual steps needed to replicate the tests. --> Created an AWS v1.29.4 cluster with the following lookup envs: ``` AMI_LOOKUP_BASEOS=rocky-9.1 AMI_LOOKUP_FORMAT=konvoy-ami-{{.BaseOS}}-release-?{{.K8sVersion}}-* AMI_LOOKUP_ORG=999867407951 ``` **Special notes for your reviewer**: <!-- Use this to provide any additional information to the reviewers. This may include: - Best way to review the PR. - Where the author wants the most review attention on. - etc. -->
|
For people coming to this issue, and trying to clarify one important issue because most of the problems are related to wrong deployment architectures that broke these deployments by this change in behavior. Since the move to IMPORTANT deploy the CLOUD-CONTROLLER-MANAGER the same as yout |
|
@aojea we were talking about this comment in the SIG cloud provider meeting today and we think it would be nice to have a more formal discussion about deployment patterns and the possible advantages and disadvantages of different styles. would you be willing to collaborate on a blog post to go into greater detail on your comment here about static pod deployment? happy to followup on slack or emails, whatever is most convenient. |
happy to collaborate, let's follow up on slack |
In k8s >= 1.29 kubelet no longer sets node internalIP when using external cloud provider kubernetes/kubernetes#121028 This regresses node startup performance in Azure CNI Overlay and Podsubnet clusters, which require the node to be assigned an internal IP before configuring pod networking. To improve node startup performance, explicitly set `--node-ip` to the IP returned from IMDS so kubelet sets the internal IP when it registers the node. If this fails, skip setting --node-ip, which is safe because cloud-node-manager will assign it later anyway.
In k8s >= 1.29 kubelet no longer sets node internalIP when using external cloud provider kubernetes/kubernetes#121028 This regresses node startup performance in Azure CNI Overlay and Podsubnet clusters, which require the node to be assigned an internal IP before configuring pod networking. To improve node startup performance, explicitly set `--node-ip` to the IP returned from IMDS so kubelet sets the internal IP when it registers the node. If this fails, skip setting --node-ip, which is safe because cloud-node-manager will assign it later anyway.
In k8s >= 1.29 kubelet no longer sets node internalIP when using external cloud provider kubernetes/kubernetes#121028 This regresses node startup performance in Azure CNI Overlay and Podsubnet clusters, which require the node to be assigned an internal IP before configuring pod networking. To improve node startup performance, explicitly set `--node-ip` to the IP returned from IMDS so kubelet sets the internal IP when it registers the node. If this fails, skip setting --node-ip, which is safe because cloud-node-manager will assign it later anyway.
In k8s >= 1.29 kubelet no longer sets node internalIP when using external cloud provider kubernetes/kubernetes#121028 This regresses node startup performance in Azure CNI Overlay and Podsubnet clusters, which require the node to be assigned an internal IP before configuring pod networking. To improve node startup performance, explicitly set `--node-ip` to the IP returned from IMDS so kubelet sets the internal IP when it registers the node. If this fails, skip setting --node-ip, which is safe because cloud-node-manager will assign it later anyway.
In k8s >= 1.29 kubelet no longer sets node internalIP when using external cloud provider kubernetes/kubernetes#121028 This regresses node startup performance in Azure CNI Overlay and Podsubnet clusters, which require the node to be assigned an internal IP before configuring pod networking. To improve node startup performance, explicitly set `--node-ip` to the IP returned from IMDS so kubelet sets the internal IP when it registers the node. If this fails, skip setting --node-ip, which is safe because cloud-node-manager will assign it later anyway.
In k8s >= 1.29 kubelet no longer sets node internalIP when using external cloud provider kubernetes/kubernetes#121028 This regresses node startup performance in Azure CNI Overlay and Podsubnet clusters, which require the node to be assigned an internal IP before configuring pod networking. To improve node startup performance, explicitly set `--node-ip` to the IP returned from IMDS so kubelet sets the internal IP when it registers the node. If this fails, skip setting --node-ip, which is safe because cloud-node-manager will assign it later anyway.
In k8s >= 1.29 kubelet no longer sets node internalIP when using external cloud provider kubernetes/kubernetes#121028 This regresses node startup performance in Azure CNI Overlay and Podsubnet clusters, which require the node to be assigned an internal IP before configuring pod networking. To improve node startup performance, explicitly set `--node-ip` to the IP returned from IMDS so kubelet sets the internal IP when it registers the node. If this fails, skip setting --node-ip, which is safe because cloud-node-manager will assign it later anyway.
In k8s >= 1.29 kubelet no longer sets node internalIP when using external cloud provider kubernetes/kubernetes#121028 This regresses node startup performance in Azure CNI Overlay and Podsubnet clusters, which require the node to be assigned an internal IP before configuring pod networking. To improve node startup performance, explicitly set `--node-ip` to the IP returned from IMDS so kubelet sets the internal IP when it registers the node. If this fails, skip setting --node-ip, which is safe because cloud-node-manager will assign it later anyway.
Kubelet, when using cloud provider external, initializes temporary the node addresses using the --node-ip flag values if set, until the cloud provider overrides it.
This behavior has undesired consequences if the cloud-provider addresses are different than the original ones, specially for hostNetwork pods, that inherit these addresses from the Node.
Since some cloud-providers depend on this behavior, in order to keep backward compatibility, assume that the specifying addresses via the node-ip flags means that the intent is to keep the existing behavior to temporary initialize the addresses.
If the node-ips are the unspecified addresses or are not set, then wait for the external cloud provider to set the node addresses.
/kind bug
/kind documentation
/kind feature
Fixes #120720