feat(kubernetes): bootstrap tenant workers with Talos via CABPT - #2610
feat(kubernetes): bootstrap tenant workers with Talos via CABPT#2610Andrei Kvapil (kvaps) wants to merge 67 commits into
Conversation
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
📝 WalkthroughWalkthroughAdds Talos bootstrap support across the chart and system providers: new Talos bootstrap CRDs/controller, cert-manager-based Talos PKI, secrets with lookup/reuse, tenant bootstrap Job, cluster template changes to emit conditional TalosConfigTemplate and Talos wiring, plus related image/Makefile/schema/helper and test updates. ChangesTalos bootstrap system
Tooling, images, schema, tests
sequenceDiagram Helm->>CertMgr: render talos-pki (create Issuer/CA/TLS cert) Estimated Code Review Effort🎯 4 (Complex) | ⏱️ ~60 minutes Suggested reviewers
Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Summary of ChangesHello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request implements Phase 1 of the Kubernetes-app split design by transitioning tenant worker node bootstrapping from Ubuntu+kubeadm to Talos Linux. The changes include integrating the Talos bootstrap provider, enhancing the Kamaji control-plane to support necessary sidecar services, and automating the lifecycle of required PKI and cluster secrets. These updates ensure that existing tenants can transition to Talos workers automatically, while new tenants will utilize Talos by default, improving overall system robustness and alignment with modern cloud-native infrastructure practices. Highlights
New Features🧠 You can now enable Memory (public preview) to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console. Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on Gemini (@gemini-code-assist) comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here. Footnotes
|
There was a problem hiding this comment.
Actionable comments posted: 3
🧹 Nitpick comments (4)
packages/system/capi-providers-cpprovider/files/control-plane-components.yaml (3)
6686-6718: 💤 Low valueConsider adding operation-specific validation rules.
The
configurationJSONPatchesschema allows any combination of fields for all operations, but RFC 6902 requires specific fields for certain operations:
fromis required formoveandcopyoperationsvalueis required foradd,replace, andtestoperations but not valid forremoveWhile the current design provides flexibility, adding CEL validation rules could catch misconfigurations at admission time rather than at runtime.
Example CEL validation rules
x-kubernetes-validations: - message: "from is required for move and copy operations" rule: "!(self.op == 'move' || self.op == 'copy') || has(self.from)" - message: "value is required for add, replace, and test operations" rule: "!(self.op == 'add' || self.op == 'replace' || self.op == 'test') || has(self.value)"Also applies to: 13836-13868
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/system/capi-providers-cpprovider/files/control-plane-components.yaml` around lines 6686 - 6718, Add CEL admission validations to the configurationJSONPatches schema to enforce RFC 6902 operation-specific required fields: ensure that when configurationJSONPatches.items.properties.op equals "move" or "copy" the "from" field must be present, and when op equals "add", "replace", or "test" the "value" field must be present (and optionally forbid "value" for "remove"); implement these as x-kubernetes-validations entries on configurationJSONPatches.items (referencing op, from, and value) so misconfigured patches are rejected at admission time.
170-14491: Clarify CRD maintenance and generation process.This CRD file contains significant schema changes (additionalServicePorts, userAnnotations, configurationJSONPatches) that align with the custom controller image version
v0.16.0-talos-csr-signer.0.Since this appears to be a custom fork of the Kamaji control-plane provider, please confirm:
- Is this CRD manually maintained or generated from the custom fork?
- Is there a process to keep it synchronized with upstream Kamaji updates?
- Should there be a comment or annotation documenting the customizations?
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/system/capi-providers-cpprovider/files/control-plane-components.yaml` around lines 170 - 14491, The CRD file contains extensive schema changes related to features like additionalServicePorts, userAnnotations, and configurationJSONPatches and matches the custom controller image version v0.16.0-talos-csr-signer.0. Please clarify if this CRD is manually maintained or automatically generated from the custom Kamaji fork, and describe the process used to keep it in sync with upstream Kamaji updates. Also, add a clear comment or annotation in the CRD file documenting that this is a customized version and specify the related controller image and version to help future maintainers understand the source and divergence from upstream.
5896-5913: ⚡ Quick winAdd schema-level pattern validation for domain-prefixed keys in userAnnotations.
The
userAnnotationsschema documents that "all keys must be domain-prefixed" but doesn't enforce this requirement via pattern validation. Users could specify non-prefixed keys that pass admission but be rejected later by the signer or cause unexpected behavior. Adding a pattern validation rule onadditionalPropertieswould catch these configuration errors at admission time.Applies to both occurrences (lines 5896-5913 and 13046-13063).
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/system/capi-providers-cpprovider/files/control-plane-components.yaml` around lines 5896 - 5913, Add schema-level validation to enforce domain-prefixed keys for the userAnnotations object: update the userAnnotations schema (the object with additionalProperties) to include a propertyNames or patternProperties rule that requires keys to match a domain-prefixed pattern (e.g. a DNS subdomain followed by a slash and a name) so invalid non-prefixed keys are rejected at admission; apply the same change to both occurrences of userAnnotations (the block describing spec.unverifiedUserAnnotations) and ensure the pattern used is consistent with Kubernetes DNS subdomain rules (for example: DNS-subdomain '/' name).packages/apps/kubernetes/templates/cluster.yaml (1)
50-50: ⚡ Quick winConsider centralizing Talos version and schematic ID.
The Talos version (
v1.13.0) and schematic ID (ce4c980...) are hardcoded in three locations: the dataVolume source URL (line 50), template variables (lines 510-511), and the machineconfig install image (line 556). If these values drift, workers would boot from one image but attempt to install from another, causing failures.♻️ Refactor to define once at template scope
{{- if not $etcd }} --- +{{- $talosVersion := "v1.13.0" }} +{{- $talosSchematic := "ce4c980550dd2ab1b17bbf2b08801c7eb59418eafe8f279833297925d67c7515" }} apiVersion: v1 kind: ConfigMapThen reference
{{ $talosVersion }}and{{ $talosSchematic }}at lines 50, 510-511, and 556, or move these tovalues.yamlfor user configurability.Also applies to: 510-511, 556-556
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/apps/kubernetes/templates/cluster.yaml` at line 50, The Talos schematic ID and version are hardcoded in multiple places (the dataVolume source URL string, the template variable block around the current schematic/version, and the machineconfig install image), so define single template-scoped variables (e.g., $talosSchematic and $talosVersion or put them in values.yaml) and replace the literal parts in the dataVolume URL (line with url), the template variable declarations (the block currently holding the schematic/version), and the machineconfig install image reference to use those variables (e.g., {{ $talosSchematic }} and {{ $talosVersion }}) so all three locations share the same source of truth.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@packages/apps/kubernetes/templates/talos/bootstrap-token-tenant-job.yaml`:
- Line 49: The kubectl image is hardcoded to alpine/k8s:1.33.4 in the
bootstrap-token-tenant Job; update it to be parameterized or documented: change
the image reference in the template for the Job (where image:
docker.io/alpine/k8s:1.33.4 is set) to use the chart value (e.g. values.version)
like other templates, or if 1.33.4 is intentionally required, add a clear
comment above that image line explaining the backward-compatibility rationale
and why the 2-minor-version skew is acceptable.
In
`@packages/system/capi-providers-cpprovider/files/control-plane-components.yaml`:
- Around line 6744-6795: The schema for additionalServicePorts lacks validation
for reserved port names and numeric port ranges: update the properties under
additionalServicePorts to add an x-kubernetes-validations rule on the name
property (e.g., disallow values 'kube-apiserver' and 'konnectivity-server'), add
minimum: 1 and maximum: 65535 to the port property, and modify targetPort's
integer anyOf branch to include minimum: 1 and maximum: 65535 while keeping the
string branch and x-kubernetes-int-or-string flag; apply the same changes to the
corresponding block referenced at the other location.
- Line 14491: The image tag in control-plane-components.yaml currently uses
ghcr.io/cozystack/cozystack/cluster-api-control-plane-provider-kamaji:v0.16.0-talos-csr-signer.0;
update this image value to use the same tag used elsewhere in the PR
(v0.16.0-cp-talos-csr-signer.1) so it matches configmaps.yaml and
providers.yaml, ensuring the image field for the
cluster-api-control-plane-provider-kamaji entry is replaced with the
v0.16.0-cp-talos-csr-signer.1 tag.
---
Nitpick comments:
In `@packages/apps/kubernetes/templates/cluster.yaml`:
- Line 50: The Talos schematic ID and version are hardcoded in multiple places
(the dataVolume source URL string, the template variable block around the
current schematic/version, and the machineconfig install image), so define
single template-scoped variables (e.g., $talosSchematic and $talosVersion or put
them in values.yaml) and replace the literal parts in the dataVolume URL (line
with url), the template variable declarations (the block currently holding the
schematic/version), and the machineconfig install image reference to use those
variables (e.g., {{ $talosSchematic }} and {{ $talosVersion }}) so all three
locations share the same source of truth.
In
`@packages/system/capi-providers-cpprovider/files/control-plane-components.yaml`:
- Around line 6686-6718: Add CEL admission validations to the
configurationJSONPatches schema to enforce RFC 6902 operation-specific required
fields: ensure that when configurationJSONPatches.items.properties.op equals
"move" or "copy" the "from" field must be present, and when op equals "add",
"replace", or "test" the "value" field must be present (and optionally forbid
"value" for "remove"); implement these as x-kubernetes-validations entries on
configurationJSONPatches.items (referencing op, from, and value) so
misconfigured patches are rejected at admission time.
- Around line 170-14491: The CRD file contains extensive schema changes related
to features like additionalServicePorts, userAnnotations, and
configurationJSONPatches and matches the custom controller image version
v0.16.0-talos-csr-signer.0. Please clarify if this CRD is manually maintained or
automatically generated from the custom Kamaji fork, and describe the process
used to keep it in sync with upstream Kamaji updates. Also, add a clear comment
or annotation in the CRD file documenting that this is a customized version and
specify the related controller image and version to help future maintainers
understand the source and divergence from upstream.
- Around line 5896-5913: Add schema-level validation to enforce domain-prefixed
keys for the userAnnotations object: update the userAnnotations schema (the
object with additionalProperties) to include a propertyNames or
patternProperties rule that requires keys to match a domain-prefixed pattern
(e.g. a DNS subdomain followed by a slash and a name) so invalid non-prefixed
keys are rejected at admission; apply the same change to both occurrences of
userAnnotations (the block describing spec.unverifiedUserAnnotations) and ensure
the pattern used is consistent with Kubernetes DNS subdomain rules (for example:
DNS-subdomain '/' name).
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: bd1a45cd-4241-413d-9d86-c6c48dc95857
⛔ Files ignored due to path filters (2)
packages/system/capi-providers-bootstrap/files/components-talos.gzis excluded by!**/*.gzpackages/system/capi-providers-cpprovider/files/components.gzis excluded by!**/*.gz
📒 Files selected for processing (14)
packages/apps/kubernetes/templates/cluster.yamlpackages/apps/kubernetes/templates/helmreleases/cilium.yamlpackages/apps/kubernetes/templates/talos/bootstrap-token-tenant-job.yamlpackages/apps/kubernetes/templates/talos/talos-pki.yamlpackages/apps/kubernetes/templates/talos/talos-secrets.yamlpackages/apps/kubernetes/tests/cluster_test.yamlpackages/apps/kubernetes/tests/kubelet-reservation_test.yamlpackages/system/capi-providers-bootstrap/files/bootstrap-components-talos.yamlpackages/system/capi-providers-bootstrap/files/metadata-talos.yamlpackages/system/capi-providers-bootstrap/templates/configmaps.yamlpackages/system/capi-providers-bootstrap/templates/providers.yamlpackages/system/capi-providers-cpprovider/files/control-plane-components.yamlpackages/system/capi-providers-cpprovider/templates/configmaps.yamlpackages/system/capi-providers-cpprovider/templates/providers.yaml
💤 Files with no reviewable changes (1)
- packages/apps/kubernetes/tests/kubelet-reservation_test.yaml
There was a problem hiding this comment.
Code Review
This pull request transitions the Kubernetes package from Kubeadm to Talos Linux bootstrapping and updates the Cluster API providers. It introduces TalosConfigTemplate, sidecar-based CSR signing, and automated PKI management. The reviewer feedback highlights several critical improvements: pinning the talos-csr-signer image version instead of using :latest, adding mandatory resource requests, limits, and security contexts for new containers, and externalizing hardcoded configurations—such as subnets, DNS domains, and Talos image metadata—into values.yaml to ensure the templates are configurable and maintainable.
myasnikovdaniil
left a comment
There was a problem hiding this comment.
Thanks for tackling this — the architectural direction is solid and the rationale comments throughout (checkStrategy: none, openstack-vs-nocloud, lookup-and-reuse, hook ordering, namespace-scoped endpoint-discovery RBAC) make the PR pleasant to read. Asking for changes on a few concrete points before merge:
Critical — destructive upgrade hazard for existing tenants
The Talos worker image is hardcoded at v1.13.0, which officially supports K8s v1.32–v1.34 only. The chart still advertises v1.30–v1.35 in the Version enum (values.yaml) and files/versions.yaml is unchanged. On helm upgrade of an existing tenant running version: v1.30 (or v1.31), the KubevirtMachineTemplate hash rolls, MachineDeployment rolls workers to TalosConfigTemplate, and new Talos workers come up running kubelet:v1.30.14 under Talos v1.13 — an unsupported combination, with no operator-facing warning. HelmRelease succeeds, customer workloads end up on a broken substrate.
This must be a chart-level hard fail, not just a release-note. Suggested guard at the top of cluster.yaml (where $talosVersion is set):
{{- $talosK8sSupportMatrix := dict
"v1.13.0" (list "v1.32" "v1.33" "v1.34")
}}
{{- $supported := index $talosK8sSupportMatrix $talosVersion }}
{{- if not (has .Values.version $supported) }}
{{- fail (printf "Kubernetes %s is not supported by Talos %s. Bump tenant 'version' to one of: %v." .Values.version $talosVersion $supported) }}
{{- end }}
Failed HelmRelease is visible in Flux/dashboards; silent rollover is not. Alternatively, consider gating this PR's Talos path behind an opt-in bootstrapMode: kubeadm|talos flag so existing tenants on v1.30/v1.31 migrate forward at their own pace. Either way, also trim the Version enum + files/versions.yaml to the Talos-supported set.
Process — 10 unanswered bot inline comments
CodeRabbit (3 actionable + 4 nitpicks) and Gemini Code Assist (7 inline) have all 10 threads with in_reply_to_id: null — zero engagement. Several findings overlap with mine below (:latest, hardcoded Talos version, hardcoded CIDRs, hardcoded dnsDomain, missing resources/securityContext on the csr-signer sidecar). Please address each thread with a fix or a brief "won't fix because X" so it's clear which findings are deliberate trade-offs.
Test coverage
tests/kubelet-reservation_test.yaml (1843 lines) was deleted, but the underlying reservation/validation logic in cluster.yaml (5%-of-memory auto-compute, 256Mi floor, 1Gi ceiling, format guards, mixed-unit-type rejection, total-reserved < effective-memory invariant) is still live and now feeds the new TalosConfigTemplate.spec.template.spec.data. Please port the assertions to match the new TCT output, or at minimum retain the failure-path tests (those don't depend on the lookup gate and would still render).
Cleanup
packages/apps/kubernetes/images/ubuntu-container-disk-v1.{30,31,32,33,34,35}.tag (6 files) are now dead — the containerDisk.image reference is gone. Delete them as part of this PR.
Inline comments below cover the per-line items (:latest image, hardcoded Talos coords, CIDRs/dnsDomain, dedup of lookups, version skew, Job retry-budget, etc.). Most of these mirror what CodeRabbit and Gemini already flagged.
Recommendation: request changes.
Per the official Talos support matrix, Talos v1.13 supports Kubernetes 1.31, 1.32, 1.33, 1.34, 1.35 and 1.36 — not just 1.32–1.34: https://docs.siderolabs.com/talos/v1.13/getting-started/support-matrix So the only entry in the chart's |
There was a problem hiding this comment.
Actionable comments posted: 4
♻️ Duplicate comments (1)
packages/apps/kubernetes/images/talos-csr-signer/Dockerfile (1)
8-8:⚠️ Potential issue | 🟠 MajorVerify the golang base image version.
The Dockerfile specifies
golang:1.26, but Go 1.26 does not exist yet (latest is approximately 1.23 as of March 2025).See the identical issue in
packages/system/capi-providers-cpprovider/images/cluster-api-control-plane-provider-kamaji/Dockerfileline 8. Both Dockerfiles should use the same valid Go version.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/apps/kubernetes/images/talos-csr-signer/Dockerfile` at line 8, The Dockerfile's base image line "FROM golang:1.26" uses a non-existent Go version; update that line to a valid released Go image (e.g., "FROM golang:1.23" or the project-standard version) and make the identical change in the other Dockerfile that currently uses "FROM golang:1.26" so both images use the same, supported Go version.
🧹 Nitpick comments (2)
packages/apps/kubernetes/templates/helmreleases/ouroboros.yaml (1)
22-43: 💤 Low valueConsider updating the comment to reflect the helper function usage.
The comment on line 22 states "Pin clusterDomain to cluster.local" but the implementation now uses a helper function. While the comment remains accurate in spirit (the helper returns
cluster.localby default), it could be slightly more precise by mentioning the helper, for example:Pin clusterDomain via the kubernetes.tenantClusterDomain helper (defaults to cluster.local).This is optional and does not affect functionality; the existing comment still correctly guides operators to the override path in lines 40-43.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/apps/kubernetes/templates/helmreleases/ouroboros.yaml` around lines 22 - 43, Update the top comment to state that clusterDomain is pinned via the kubernetes.tenantClusterDomain helper (which defaults to "cluster.local") instead of asserting a hardcoded value; e.g. replace "Pin clusterDomain to cluster.local" with "Pin clusterDomain via the kubernetes.tenantClusterDomain helper (defaults to cluster.local)". Keep the existing guidance about overriding via addons.ouroboros.valuesOverride.ouroboros.controller.clusterDomain and preserve the explanatory context about tenant CoreDNS and kubelet behavior.packages/system/capi-providers-cpprovider/images/cluster-api-control-plane-provider-kamaji/Dockerfile (1)
19-20: ⚖️ Poor tradeoffConsider adding checksum verification for the downloaded source tarball.
The Dockerfile downloads upstream source from GitHub without verifying its integrity. An attacker who compromises the upstream repository or performs a man-in-the-middle attack could inject malicious code into the build.
Consider pinning a checksum (SHA256) alongside the
COMMITARG and verifying it:ARG COMMIT=e52ee00a9f5fcb516314ab04da0c668a3dfaca54 ARG COMMIT_SHA256=<expected_sha256_of_tarball> RUN curl -sSL "https://github.com/clastix/cluster-api-control-plane-provider-kamaji/archive/${COMMIT}.tar.gz" \ -o source.tar.gz && \ echo "${COMMIT_SHA256} source.tar.gz" | sha256sum -c - && \ tar -xzf source.tar.gz --strip=1 && \ rm source.tar.gzThis adds supply-chain security at the cost of maintaining the checksum when bumping
COMMIT.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/system/capi-providers-cpprovider/images/cluster-api-control-plane-provider-kamaji/Dockerfile` around lines 19 - 20, Add checksum verification when downloading the upstream tarball in the Dockerfile: introduce an ARG for the expected SHA256 (e.g., COMMIT_SHA256) alongside COMMIT, download the tarball to a temporary file instead of streaming, verify it with sha256sum -c using the COMMIT_SHA256, only then extract with tar --strip=1 and remove the temp file; update the RUN that references COMMIT to perform these steps and fail the build if the checksum check fails.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@packages/apps/kubernetes/images/talos-csr-signer/Dockerfile`:
- Around line 13-14: The Dockerfile declares build args TARGETOS and TARGETARCH
without defaults causing builds to fail when not supplied; update the ARG
declarations (both the first occurrences and the later ones around lines showing
21-22) to provide sensible defaults (e.g., set TARGETOS=linux and
TARGETARCH=amd64) so go build produces a correct binary when build args are
omitted; ensure you change ARG TARGETOS and ARG TARGETARCH to include the
default values and keep the same argument names used in the go build invocation.
In `@packages/apps/kubernetes/templates/cluster.yaml`:
- Line 644: Inside the nodeGroups range the dot (.) is the nodeGroup object, so
calling include "kubernetes.tenantClusterDomain" . passes the wrong context;
change the include to use the root context (e.g., include
"kubernetes.tenantClusterDomain" $) so the tenantClusterDomain template receives
the chart/global root context instead of the nodeGroup object.
In `@packages/apps/kubernetes/templates/talos/bootstrap-token-tenant-job.yaml`:
- Around line 55-58: The KUBECONFIG/TENANT_KUBECONFIG env vars point to
/tenant-admin/admin.svc but the secret key used by the control-plane is
super-admin.svc; update the environment values so both TENANT_KUBECONFIG and
KUBECONFIG reference /tenant-admin/super-admin.svc (or change to the actual
secret key name used by the control-plane) so kubectl can load the correct
kubeconfig when the job (env names TENANT_KUBECONFIG and KUBECONFIG) runs.
In
`@packages/system/capi-providers-cpprovider/images/cluster-api-control-plane-provider-kamaji/Dockerfile`:
- Around line 14-15: The Dockerfile declares ARG TARGETOS and ARG TARGETARCH but
only TARGETOS has a fallback; make defaults consistent by declaring ARG
TARGETOS=linux and ARG TARGETARCH=amd64 (or another desired arch) so builds work
when variables are unset, and then update the build invocation (the RUN that
calls go build) to reference ${TARGETOS} and ${TARGETARCH} directly (removing
inline shell fallbacks) so the ARG defaults are authoritative; locate the ARG
lines and the RUN go build step in the Dockerfile to apply the change.
---
Duplicate comments:
In `@packages/apps/kubernetes/images/talos-csr-signer/Dockerfile`:
- Line 8: The Dockerfile's base image line "FROM golang:1.26" uses a
non-existent Go version; update that line to a valid released Go image (e.g.,
"FROM golang:1.23" or the project-standard version) and make the identical
change in the other Dockerfile that currently uses "FROM golang:1.26" so both
images use the same, supported Go version.
---
Nitpick comments:
In `@packages/apps/kubernetes/templates/helmreleases/ouroboros.yaml`:
- Around line 22-43: Update the top comment to state that clusterDomain is
pinned via the kubernetes.tenantClusterDomain helper (which defaults to
"cluster.local") instead of asserting a hardcoded value; e.g. replace "Pin
clusterDomain to cluster.local" with "Pin clusterDomain via the
kubernetes.tenantClusterDomain helper (defaults to cluster.local)". Keep the
existing guidance about overriding via
addons.ouroboros.valuesOverride.ouroboros.controller.clusterDomain and preserve
the explanatory context about tenant CoreDNS and kubelet behavior.
In
`@packages/system/capi-providers-cpprovider/images/cluster-api-control-plane-provider-kamaji/Dockerfile`:
- Around line 19-20: Add checksum verification when downloading the upstream
tarball in the Dockerfile: introduce an ARG for the expected SHA256 (e.g.,
COMMIT_SHA256) alongside COMMIT, download the tarball to a temporary file
instead of streaming, verify it with sha256sum -c using the COMMIT_SHA256, only
then extract with tar --strip=1 and remove the temp file; update the RUN that
references COMMIT to perform these steps and fail the build if the checksum
check fails.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: 7640461c-29ab-4337-ae7c-056bcf5a08c1
⛔ Files ignored due to path filters (1)
packages/system/capi-providers-cpprovider/files/components.gzis excluded by!**/*.gz
📒 Files selected for processing (29)
Makefileapi/apps/v1alpha1/kubernetes/types.gopackages/apps/kubernetes/Makefilepackages/apps/kubernetes/README.mdpackages/apps/kubernetes/files/versions.yamlpackages/apps/kubernetes/images/kubectl.tagpackages/apps/kubernetes/images/talos-csr-signer.tagpackages/apps/kubernetes/images/talos-csr-signer/Dockerfilepackages/apps/kubernetes/images/ubuntu-container-disk-v1.30.tagpackages/apps/kubernetes/images/ubuntu-container-disk-v1.31.tagpackages/apps/kubernetes/images/ubuntu-container-disk-v1.32.tagpackages/apps/kubernetes/images/ubuntu-container-disk-v1.33.tagpackages/apps/kubernetes/images/ubuntu-container-disk-v1.34.tagpackages/apps/kubernetes/images/ubuntu-container-disk-v1.35.tagpackages/apps/kubernetes/images/ubuntu-container-disk/Dockerfilepackages/apps/kubernetes/templates/_helpers.tplpackages/apps/kubernetes/templates/cluster.yamlpackages/apps/kubernetes/templates/helmreleases/ouroboros.yamlpackages/apps/kubernetes/templates/talos/bootstrap-token-tenant-job.yamlpackages/apps/kubernetes/templates/talos/talos-pki.yamlpackages/apps/kubernetes/tests/kubelet_reservation_test.yamlpackages/apps/kubernetes/values.schema.jsonpackages/apps/kubernetes/values.yamlpackages/system/capi-providers-cpprovider/Makefilepackages/system/capi-providers-cpprovider/files/control-plane-components.yamlpackages/system/capi-providers-cpprovider/images/cluster-api-control-plane-provider-kamaji/Dockerfilepackages/system/capi-providers-cpprovider/templates/configmaps.yamlpackages/system/capi-providers-cpprovider/templates/providers.yamlpackages/system/kubernetes-rd/cozyrds/kubernetes.yaml
💤 Files with no reviewable changes (8)
- packages/apps/kubernetes/images/ubuntu-container-disk-v1.35.tag
- packages/apps/kubernetes/images/ubuntu-container-disk-v1.31.tag
- packages/apps/kubernetes/images/ubuntu-container-disk-v1.32.tag
- packages/apps/kubernetes/images/ubuntu-container-disk-v1.33.tag
- packages/apps/kubernetes/images/ubuntu-container-disk-v1.34.tag
- packages/apps/kubernetes/images/ubuntu-container-disk-v1.30.tag
- packages/apps/kubernetes/files/versions.yaml
- packages/apps/kubernetes/images/ubuntu-container-disk/Dockerfile
✅ Files skipped from review due to trivial changes (3)
- packages/apps/kubernetes/images/talos-csr-signer.tag
- packages/apps/kubernetes/images/kubectl.tag
- packages/apps/kubernetes/templates/_helpers.tpl
🚧 Files skipped from review as they are similar to previous changes (1)
- packages/system/capi-providers-cpprovider/templates/configmaps.yaml
There was a problem hiding this comment.
♻️ Duplicate comments (1)
packages/apps/kubernetes/templates/cluster.yaml (1)
644-644:⚠️ Potential issue | 🟠 Major | ⚡ Quick win
dnsDomainis rendered inside therange $groupName, $group := .Values.nodeGroupsloop, so.is the nodeGroup, not the chart root.The
kubernetes.tenantClusterDomainhelper expects the chart root context (it reads.Values.…). With.being the nodeGroup map, the helper either fails or silently resolves to an empty/default cluster domain, which would mean Talos workers configure the kubelet with the wrong DNS domain and DNS lookups against*.svc.<tenantClusterDomain>would diverge from what Kamaji/CoreDNS expects. Use$(root) here, consistent with line 293 where.happens to equal$because it's outside the range.🐛 Proposed fix
- dnsDomain: {{ include "kubernetes.tenantClusterDomain" . }} + dnsDomain: {{ include "kubernetes.tenantClusterDomain" $ }}🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/apps/kubernetes/templates/cluster.yaml` at line 644, The dnsDomain value is being rendered inside the range loop so "." is the nodeGroup rather than the chart root, causing the helper kubernetes.tenantClusterDomain to read the wrong context; update the template to call the helper with the root context (use $) instead of . (i.e., replace the current dnsDomain: {{ include "kubernetes.tenantClusterDomain" . }} with a call that passes $), ensuring the helper reads .Values from the chart root when rendering dnsDomain for the nodeGroup.
🧹 Nitpick comments (3)
packages/system/capi-providers-cpprovider/images/cluster-api-control-plane-provider-kamaji/Dockerfile (1)
22-23: ⚡ Quick winAdd
--failto curl and enable pipefail so download errors abort the build.Without
--fail,curl -sSLswallows non-2xx HTTP responses; combined with noset -o pipefailon the default/bin/sh -c, the RUN can silently succeed even when the upstream tag fetch fails or the body is empty. The downstreamgit applyandgo buildwould then run on whatever happens to be in the working dir (or hard-fail with a confusing diagnostic). Belt-and-braces this so the failure surface is the actual curl error.♻️ Proposed fix
-RUN curl -sSL "https://github.com/clastix/cluster-api-control-plane-provider-kamaji/archive/refs/tags/${VERSION}.tar.gz" \ - | tar -xzf - --strip=1 +SHELL ["/bin/sh", "-eo", "pipefail", "-c"] +RUN curl --fail -sSL "https://github.com/clastix/cluster-api-control-plane-provider-kamaji/archive/refs/tags/${VERSION}.tar.gz" \ + | tar -xzf - --strip=1🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/system/capi-providers-cpprovider/images/cluster-api-control-plane-provider-kamaji/Dockerfile` around lines 22 - 23, The RUN that streams the tarball via curl into tar should be hardened: add curl's --fail flag and enable pipefail so that non-2xx responses or broken pipes cause the Docker build to fail. Update the RUN that uses curl (the line invoking curl -sSL "...${VERSION}..." | tar -xzf - --strip=1) to run in a shell that sets -o pipefail (and -e) and to pass --fail to curl; this ensures errors are surfaced before subsequent commands like git apply or go build run on bad input.packages/system/capi-providers-cpprovider/Makefile (2)
43-44: ⚡ Quick winUse
curl --failso a brokenUPSTREAM_VERSIONdoesn't pass through totar.Same pattern as the Dockerfile:
curl -sSLsucceeds on HTTP errors and pipes whatever body it got into tar. If a future bump renames a tag or the registry hiccups, you'll get a tar decompression error instead of a clear "release not found" message, andmake updatemay go on to commit/apply patches against partial sources.♻️ Proposed fix
- curl -sSL "https://github.com/clastix/cluster-api-control-plane-provider-kamaji/archive/refs/tags/$(UPSTREAM_VERSION).tar.gz" \ - | tar -xzf - --strip=1 -C .update-tmp + set -o pipefail; curl --fail -sSL "https://github.com/clastix/cluster-api-control-plane-provider-kamaji/archive/refs/tags/$(UPSTREAM_VERSION).tar.gz" \ + | tar -xzf - --strip=1 -C .update-tmp🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/system/capi-providers-cpprovider/Makefile` around lines 43 - 44, The curl invocation in the Makefile (the line invoking curl -sSL "https://github.com/.../$(UPSTREAM_VERSION).tar.gz" | tar -xzf - --strip=1 -C .update-tmp) should add the --fail flag so HTTP errors cause curl to exit non‑zero instead of streaming an error page into tar; update that curl command to include --fail (or -f) alongside -sSL to ensure make update fails immediately when the release/tag is not found.
27-33: 💤 Low valueFold the metadata-JSON cleanup into the shell chain so failures don't leave stale artifacts.
Recipe lines 27–32 run in one shell; line 33 is a separate sub-shell that only fires when the previous line succeeded. If the digest/sed/gzip chain fails (e.g., the
containerimage.digestkey isn't in the metadata), the JSON sticks around and the nextmake imagemay pick up a stale value. Moving the cleanup inside the chain (or trapping on EXIT) makes the target idempotent.♻️ Proposed fix
digest=$$(yq e '."containerimage.digest"' images/cluster-api-control-plane-provider-kamaji.json -o json -r); \ + trap 'rm -f images/cluster-api-control-plane-provider-kamaji.json' EXIT; \ IMG="$(IMAGE_BASE):$(IMAGE_TAG)@$$digest"; \ echo "$$IMG" > images/cluster-api-control-plane-provider-kamaji.tag; \ sed -i.bak -E "s|image: ghcr\.io/.+/cluster-api-control-plane-provider-kamaji:[^[:space:]]+|image: $$IMG|" files/control-plane-components.yaml; \ rm files/control-plane-components.yaml.bak; \ gzip -nc files/control-plane-components.yaml > files/components.gz - rm -f images/cluster-api-control-plane-provider-kamaji.json🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/system/capi-providers-cpprovider/Makefile` around lines 27 - 33, The Makefile target leaves images/cluster-api-control-plane-provider-kamaji.json behind if the digest/sed/gzip chain fails because the final rm is executed in a separate shell; fold the cleanup into the same shell chain or add an EXIT trap so removal always runs. Update the block that defines digest=$$(yq e ...), IMG=... and the subsequent sed/gzip pipeline (the commands manipulating images/cluster-api-control-plane-provider-kamaji.json and files/control-plane-components.yaml) to either append the rm images/cluster-api-control-plane-provider-kamaji.json into that same chained line (so it runs even on failures using ; or || true as appropriate) or install a trap 'rm -f images/cluster-api-control-plane-provider-kamaji.json' at the start of the shell so the JSON is removed on exit; ensure you reference the exact symbols: digest variable, IMG variable, sed -i.bak invocation, gzip -nc, and rm images/cluster-api-control-plane-provider-kamaji.json so the cleanup is always executed.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Duplicate comments:
In `@packages/apps/kubernetes/templates/cluster.yaml`:
- Line 644: The dnsDomain value is being rendered inside the range loop so "."
is the nodeGroup rather than the chart root, causing the helper
kubernetes.tenantClusterDomain to read the wrong context; update the template to
call the helper with the root context (use $) instead of . (i.e., replace the
current dnsDomain: {{ include "kubernetes.tenantClusterDomain" . }} with a call
that passes $), ensuring the helper reads .Values from the chart root when
rendering dnsDomain for the nodeGroup.
---
Nitpick comments:
In
`@packages/system/capi-providers-cpprovider/images/cluster-api-control-plane-provider-kamaji/Dockerfile`:
- Around line 22-23: The RUN that streams the tarball via curl into tar should
be hardened: add curl's --fail flag and enable pipefail so that non-2xx
responses or broken pipes cause the Docker build to fail. Update the RUN that
uses curl (the line invoking curl -sSL "...${VERSION}..." | tar -xzf -
--strip=1) to run in a shell that sets -o pipefail (and -e) and to pass --fail
to curl; this ensures errors are surfaced before subsequent commands like git
apply or go build run on bad input.
In `@packages/system/capi-providers-cpprovider/Makefile`:
- Around line 43-44: The curl invocation in the Makefile (the line invoking curl
-sSL "https://github.com/.../$(UPSTREAM_VERSION).tar.gz" | tar -xzf - --strip=1
-C .update-tmp) should add the --fail flag so HTTP errors cause curl to exit
non‑zero instead of streaming an error page into tar; update that curl command
to include --fail (or -f) alongside -sSL to ensure make update fails immediately
when the release/tag is not found.
- Around line 27-33: The Makefile target leaves
images/cluster-api-control-plane-provider-kamaji.json behind if the
digest/sed/gzip chain fails because the final rm is executed in a separate
shell; fold the cleanup into the same shell chain or add an EXIT trap so removal
always runs. Update the block that defines digest=$$(yq e ...), IMG=... and the
subsequent sed/gzip pipeline (the commands manipulating
images/cluster-api-control-plane-provider-kamaji.json and
files/control-plane-components.yaml) to either append the rm
images/cluster-api-control-plane-provider-kamaji.json into that same chained
line (so it runs even on failures using ; or || true as appropriate) or install
a trap 'rm -f images/cluster-api-control-plane-provider-kamaji.json' at the
start of the shell so the JSON is removed on exit; ensure you reference the
exact symbols: digest variable, IMG variable, sed -i.bak invocation, gzip -nc,
and rm images/cluster-api-control-plane-provider-kamaji.json so the cleanup is
always executed.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: fd70aea8-5fd5-47f9-b98c-452c9062c5b0
⛔ Files ignored due to path filters (1)
packages/system/capi-providers-cpprovider/files/components.gzis excluded by!**/*.gz
📒 Files selected for processing (7)
packages/apps/kubernetes/templates/cluster.yamlpackages/system/capi-providers-cpprovider/Makefilepackages/system/capi-providers-cpprovider/files/control-plane-components.yamlpackages/system/capi-providers-cpprovider/images/cluster-api-control-plane-provider-kamaji/Dockerfilepackages/system/capi-providers-cpprovider/images/cluster-api-control-plane-provider-kamaji/patches/0001-backport-additional-service-ports-pr347.diffpackages/system/capi-providers-cpprovider/templates/configmaps.yamlpackages/system/capi-providers-cpprovider/templates/providers.yaml
What about other changes that were requested? |
0d30d9d to
a86f300
Compare
There was a problem hiding this comment.
Re-review. Most of the prior feedback is addressed — Talos↔K8s support-matrix guard, v1.30 removed from Version enum/files/versions.yaml/kubernetes-rd CRD schema, kamaji control-plane provider rebuilt from upstream v0.19.0 (image digest-pinned, version labels aligned), talos-csr-signer sidecar hardened (resources + non-root/RO-rootfs/drop-all-caps/seccomp), talos.version/talos.schematicID exposed as values, pod/service CIDRs collapsed into top-of-template constants, four duplicated lookups hoisted above the nodeGroup loop, images.kubectl override + tightened bootstrap-token Job retry budget, wipe: false annotated, dead ubuntu-container-disk build chain removed, every bot thread now has a reply. Destructive-upgrade hazard from the last pass is gone.
Two non-blocking nits worth folding into a follow-up:
packages/apps/kubernetes/templates/talos/bootstrap-token-tenant-job.yaml:55-58—TENANT_KUBECONFIGenv var is set but never read (onlyKUBECONFIGmatters to kubectl, and the inline shell doesn't reference it). May be drop it?
One acknowledged trade-off: tests/kubelet_reservation_test.yaml keeps the failure-path coverage but the happy-path TCT assertions from the kubeadm-era file weren't ported (the in-file comment explains why — TCT data is a YAML-encoded string, Secret mocking is left as a follow-up). Please open an issue so this doesn't drop off the radar.
LGTM.
f236a18 to
ef5eb0a
Compare
The TalosConfigTemplate rendered by the kubernetes app chart had its
fields placed directly under spec instead of under spec.template.spec.
CABPT (bootstrap.cluster.x-k8s.io/v1alpha3) requires the standard
CAPI template wrapper and rejects the resource with:
TalosConfigTemplate.bootstrap.cluster.x-k8s.io
"<release>-<group>" is invalid: spec.template: Required value
This was introduced when the original Talos-bootstrap commit replaced
the KubeadmConfigTemplate block by removing the kubeadm spec.template
wrapper but did not re-add the equivalent for the new resource. Helm
template still rendered the document because YAML accepts the
fields-under-spec shape as valid YAML, but the API server admission
rejects it on apply. Manifested as: HelmRelease succeeds installing
every other resource, TalosConfigTemplate creation fails, CAPI never
creates Machines, no Talos worker boots.
Caught while exercising the full Phase 1 install path end-to-end on
a dev cluster: the chart-rendered manifest applies cleanly until CABPT
admission, which rolls the HR back. With the wrapper restored, TCT
admits successfully and the MachineDeployment.bootstrap.configRef
resolves to a valid template.
Existing helm unittest suite did not catch this because it only
asserts on TCT presence and on individual field values via
documentIndex, not on the resource's spec.template invariant. A
follow-up assertion `spec.template.spec.generateType == none` would
prevent regressions.
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
The test script (tests/containerd-config-path-test.sh) plus its two
toml fixtures (tests/fixtures/containerd-{1x,2x}.toml) guarded a pair
of sed lines that lived inside the kubeadm-era KubeadmConfigTemplate's
preKubeadmCommands. That whole block disappeared when the chart
switched to TalosConfigTemplate — there are no preKubeadmCommands in
the Talos worker bootstrap path, containerd is configured by Talos
itself via its machineconfig dialect (machine.files / machine.kubelet
extraConfig), not by post-boot shell sed.
After this PR the script's grep -qF against the template fails with
`sed line drifted` because the strings it's looking for are simply
gone, taking down the chart's `make test` and the repo-level
`helm-unit-tests` CI step.
Remove the script, its Makefile invocation, and the now-orphan
fixtures directory. If a Talos-equivalent containerd assertion is
desired later it belongs in helm unittest against the rendered
machineconfig blob, not as a regex check on cluster.yaml.
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
The image-talos-csr-signer target still used the legacy `$(call settag, ...)` helper, which was removed from hack/common-envs.mk in main and replaced by the `$(call image-tags, <repo>, <versioned-tag>)` macro used by every other image-* target in this Makefile. After rebasing onto current main the unresolved `settag` call expands to an empty string, so the build emits `--tag <REGISTRY>/talos-csr-signer:` with nothing after the colon and docker buildx rejects it with `invalid reference format`. CI Build job exits 1. Re-aligning with image-cluster-autoscaler / image-kubevirt-csi-driver which already use image-tags. The follow-up `echo` line that pins the digest into images/talos-csr-signer.tag now references $(IMAGE_TAG) directly, matching the same convention. Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
…idecar boot The e2e fixture's `kubectl wait` budgets were tuned to the pre-Talos Kamaji cold-start (3-4 minutes for kube-apiserver+controller-manager+ scheduler+konnectivity). After this PR the apiserver pod additionally pulls and starts the talos-csr-signer sidecar, and cert-manager must issue three Talos PKI Certificates (talos-ca, talos-tls-cert, plus the Kamaji-issued ca) before the wait-for-kubeconfig init container clears. Cold-start in a fresh CI sandbox (no warm image cache) consistently crosses the old 4m/5m budgets and the test fails at `kubectl wait tcp ... --timeout=5m` with `error: timed out waiting for the condition on tenantcontrolplanes/<name>`. Raise both `TenantControlPlaneCreated` and `tcp ... version.status=Ready` budgets to 10m, matching the `release.cozystack.io/helm-install-timeout: 15m` that cozystack-api stamps onto every Kubernetes tenant HR. Downstream waits (`deploy --timeout=4m`, `machinedeployment --timeout=1m`) are unaffected by Talos PKI work and keep their tighter budgets. Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
…admission accepts the unmodified install
The MachineHealthCheck CRD's `spec.maxUnhealthy` is an IntOrString. The
CAPI admission webhook accepts either a bare integer ("0", "1", ...) or
a percentage ("0%", "50%") but rejects a bare numeric string with:
admission webhook "validation.machinehealthcheck.cluster.x-k8s.io"
denied the request: MachineHealthCheck ... is invalid:
spec.maxUnhealthy: Invalid value: intstr.IntOrString{...,
StrVal:"0"}: must be either an int or a percentage:
invalid value for IntOrString: invalid type: string is not a
percentage
The previous default ("0", a string without a percent sign) hit exactly
that path and caused Helm install to fail on every fresh tenant
Kubernetes install — surfaced by the e2e suite's
`Create a tenant Kubernetes control plane with latest version` step,
which then dropped the chart (cascade-deleted KamajiControlPlane) and
the downstream wait never had anything to find.
"0%" is semantically identical (any unhealthy node triggers remediation
immediately) and passes admission. Custom values like "50%" or bare
integers like "1" continue to work; the field stays parametric for
operators that want to relax remediation during a kubeadm-to-Talos
rollover.
Regenerated values.schema.json, README.md, types.go, and the
kubernetes-rd cozyrds OpenAPI schema via the chart's `make generate`.
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
PR review on this branch flagged that the previous default ("0" string)
was still wrong even after the values.yaml default flip: any operator
that overrides nodeHealthCheck.maxUnhealthy to a bare integer (e.g. "1",
which is also a valid IntOrString) gets the same admission rejection,
because `| quote` unconditionally wraps the rendered value in quotes
and the MHC webhook treats "1" as a non-percentage string.
Render conditionally: if the supplied value has a `%` suffix, keep the
existing quoted-string form (percentages must be strings). Otherwise,
render as a bare integer via the Sprig `int` function — that handles
the chart-default "0%", the now-safe "0", and any operator override
like "1" or "5" all in one path.
Follow-up to PR review:
cozystack#2610 (comment)
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
… Helm template into post-install hook Job
Chart used Helm `lookup` to gate the TalosConfigTemplate (TCT) and to
fold the apiserver Service ClusterIP into KamajiControlPlane's
network.certSANs. Both inputs — the Service, the Talos-CA TLS Secret,
the Kamaji-issued tenant CA Secret — only exist AFTER Helm applies the
first render: Kamaji controller emits the Service in reaction to the
KamajiControlPlane CR this same chart applies, cert-manager issues the
Talos PKI from the Certificate CR this same chart applies, and Kamaji
generates the tenant CA as part of its bootstrap. On a fresh tenant
install all three lookups return nil on first render, $talosReady is
false, TCT is skipped, and the certSANs list omits the ClusterIP.
Helm 3's release-storage three-way merge then refuses to upgrade
because no values changed. The lookups become valid 30s later, but
nothing requests a re-render. The MachineDeployment.bootstrap.configRef
points at a TalosConfigTemplate that never lands, so CAPI's
MachineDeployment controller emits `cannot create a new MachineSet
when templates do not exist` forever. The apiserver TLS cert is
generated by Kamaji without the live ClusterIP in its SAN list, so
even if a worker were to boot, its kubelet would reject the apiserver
on `x509: certificate is valid for <DNS> not <IP>`.
Suspend+resume and values-drift tricks force a second render manually
but every fresh install hits the same race; that's not a chart-level
fix, it's a workaround the operator has to remember on every tenant.
Move both objects into a post-install/post-upgrade Helm hook Job
(templates/talos/talos-reconcile-job.yaml) that:
- waits up to ~10m for Service.spec.clusterIP, <release>-talos-ca,
<release>-ca to be observable;
- JSON-merge-patches KamajiControlPlane.spec.network.certSANs to
include the live ClusterIP plus the two Service DNS names (Kamaji
re-issues the apiserver TLS cert in response);
- reads stable random tokens from <release>-talos-secrets (chart-self
rendered, always present at hook time);
- kubectl applies the TalosConfigTemplate with the full machineconfig
inlined and an ownerReference to the KamajiControlPlane.
The owner reference lets the Kubernetes garbage collector reap the TCT
when the chart's KCP is deleted on `helm uninstall`; no pre-delete
hook needed. The Job runs at hook-weight 5, between the chart's
post-install resources and the bootstrap-token-tenant-job at weight 10,
so the bootstrap-token Secret lands inside the tenant kube-system only
after the worker machineconfig that consumes it is in place.
cluster.yaml drops the lookup gates ($apiSvc / $apiSvcIP /
$talosSecrets / $talosCA / $k8sCA / $talosReady / $talosToken /
$clusterId / $clusterSecret / $bootstrapToken / $talosCACrt /
$k8sCACrt / $apiEndpoint) and the gated TCT block. The certSANs list
on the KamajiControlPlane keeps only the two DNS entries; the
ClusterIP arrives via the hook.
The e2e fixture (hack/e2e-apps/run-kubernetes.sh) waits 5m instead of
1m for `machinedeployment ... status.replicas=2` so the
post-install-hook + Talos image pull + first VM boot fit in budget on
a cold sandbox.
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
… reservations The post-install hook Job is the new home of the TalosConfigTemplate authorship (replacing the lookup-and-reuse pattern). On any cozystack tenant the default-deny CiliumClusterwideNetworkPolicy on the release namespace blocks the hook pod from reaching the management apiserver to read the Service ClusterIP / Talos CA / k8s CA / KamajiControlPlane it has to reconcile, leaving the Job stuck in a wait loop and the release pending-install. Ship a CiliumNetworkPolicy alongside the hook that allows egress to the kube-apiserver entity (and kube-dns for name resolution) for pods labelled cozystack.io/talos-reconcile=<release>. Cilium policies are additive, so the tenant default-deny stays in force for everything else in the namespace. Inline the kubelet-reservation computation from cluster.yaml directly in the hook template; the previous helper reference was unresolved at render time and aborted the chart. Numbers match cluster.yaml's auto-computed system/kube reserved memory (5% of effective memory clamped to 256Mi..1Gi) and CPU (5% clamped to 50m..500m). Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
…hook The Talos PKI Certificate that secures the trustd / apid sidecar endpoint at https://<Service ClusterIP>:50001 used the same lookup-and-reuse pattern as the TalosConfigTemplate the previous fix already moved out into the post-install hook. On a fresh install the Service ClusterIP does not yet exist when the chart renders, the lookup returns nil, and the Certificate is issued with only the DNS-name SANs. Workers then fail TLS to apid with "cannot validate certificate for <ClusterIP> because it doesn't contain any IP SANs" and stall at "service apid to be up". Drop the lookup gate from talos-pki.yaml and add a second kubectl patch step to the talos-reconcile hook Job that writes spec.ipAddresses on <release>-talos-tls-cert once the Service is live. cert-manager reissues the cert with the new SAN, the worker's TLS handshake succeeds, and apid comes up. Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
…los worker migration
Seven distinct issues found during cozy-review and end-to-end testing
on a dev cluster. Bundled here as a single squashed commit:
* nodeHealthCheck.maxUnhealthy default 0% -> 50%.
Talos workers boot in pulls from factory.talos.dev that exceed the
10m nodeStartupTimeout on slow networks. With the prior 0% default a
single Machine briefly NotReady triggers MHC remediation, CAPI spawns
a replacement that also boots slowly, and the cluster loops. 50%
leaves room for transient unhealthy nodes during the kubeadm-to-Talos
rollover; drop back to 0% in a follow-up once fleets stabilise.
* bootstrap-token Job backoffLimit 5 -> 10.
Aligns the Job's worst-case wall time (~15m under exponential backoff
capped at 6m per retry) with the chart's HR install timeout
(release.cozystack.io/helm-install-timeout: 15m). At backoffLimit=5
the Job could exhaust retries before the tenant apiserver came up on
a slow cluster, failing the chart upgrade silently.
* migration 44 (v1.30 -> v1.31) surfaces per-tenant failures.
Tracks failed (ns, name) pairs and writes them to a
cozystack.io/migration-44-failed-tenants annotation on the
cozystack-version ConfigMap. Operators previously had to grep Job
logs because the migration exits 0 (intentional, for retry) and the
Job status stays green while individual tenants are stuck at v1.30.
* talos-reconcile-job memory limit 64Mi -> 256Mi.
Reproduced OOMKill on dev3: pod completed the wait loop and the
KamajiControlPlane / Certificate JSON-merge patches, then died at
the kubectl-apply that pipes a TalosConfigTemplate manifest. Fresh
installs survive the 64Mi limit because kubectl's working set is
smaller against an empty cluster; upgrades on established tenants
pay the merge-and-diff cost. 256Mi leaves headroom for the embedded
JSON marshalling and discovery cache.
* Talos workers reach the apiserver via DNS instead of the live
ClusterIP. The post-install hook previously patched the Talos PKI
Certificate to add the apiserver Service ClusterIP as an IP SAN.
cert-manager reissues the Secret but the csr-signer sidecar loads
its TLS cert once at startup and does not watch the file, so on a
fresh install it keeps serving the original (no-IP-SAN) cert.
Talos apid then fails the trustd TLS handshake with 'x509: cannot
validate certificate for <IP> because it does not contain any IP
SANs' and the worker boot stalls on 'service apid to be up'
indefinitely. Drop the certificate patch and route Talos workers
to the apiserver via <release>.<namespace>.svc. The Talos
machineconfig now sets cluster.controlPlane.endpoint to the DNS
form and seeds machine.network.extraHostEntries with the live
ClusterIP, so worker /etc/hosts resolves the name without depending
on tenant CoreDNS (which is itself bootstrapped after Cilium, which
dials this same endpoint). The Talos PKI Certificate already carries
DNS SANs from chart render time so TLS verification succeeds against
the chart-issued cert with no race. Cilium's k8sServiceHost can also
drop the lookup-and-reuse fallback and use plain DNS.
* CABPT controller-manager runs under a uniquely-named ServiceAccount.
The vendored CABPT components manifest uses the implicit 'default'
SA in cabpt-system. cluster-api operator copies the manifest and
rewrites every {kind: ServiceAccount, name: default, namespace:
cabpt-system} subject to point at the BootstrapProvider's own
namespace (cozy-cluster-api). The Deployment still runs as
cabpt-system/default, the controller has no RBAC and CrashLoops on
'cannot list resource ... at the cluster scope'. TalosConfigs never
reconcile and Machines stay forever in WaitingForBootstrapData. The
kubeadm bootstrap provider in the same chart works because it uses
a uniquely-named SA the operator leaves alone. Mirror that pattern
for CABPT: ServiceAccount cabpt-manager in cabpt-system, wire all
four (Cluster)RoleBindings to it, and set the Deployment's
serviceAccountName accordingly. Regenerated components-talos.gz
from the edited source.
* Drop the GPU node-label test obsoleted by the Talos worker switch.
Main added a test asserting GPU node-groups register their nodes
with gpu=on via KubeadmConfigTemplate.spec.template.spec.joinConfiguration.nodeRegistration.kubeletExtraArgs.node-labels.
After this PR, KubeadmConfigTemplate is replaced by
TalosConfigTemplate rendered out-of-band by the talos-reconcile
hook and the chart no longer carries the kubelet args the test
inspects. The GPU labeling feature itself is not yet wired up on
Talos workers in this Phase 1 (group.gpus still attaches GPU
devices to the VM but nothing sets gpu=on on the resulting node).
Tracking as a Phase 1 regression rather than restoring the
now-irrelevant kubeadm-shaped assertion.
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
…olver Tenant Kubernetes workers in main run as Ubuntu+kubeadm KubeVirt VMs and inherit pod DNS via cloud-init, so they resolve management-cluster services (NFS endpoints used by kubevirt-csi like linstor-csi-nfs.cozy-linstor.svc) out of the box. Talos workers in this PR are strict and consult only what is in machineconfig; with no host resolver wired up, kubelet's NFS mount syscall fails with NXDOMAIN when kubevirt-csi-backed PVCs are mounted. The reconcile hook now looks up the management CoreDNS Service ClusterIP at runtime and injects it into machine.network.nameservers in the TalosConfigTemplate. Tenant in-cluster pods are unaffected: they use kubelet --cluster-dns (tenant CoreDNS) via dnsPolicy: ClusterFirst. This nameserver only covers host-side paths (mounts, image pulls, pods that opt out of cluster DNS), which is what main provided via cloud-init. Adds a narrow ClusterRole + ClusterRoleBinding scoped to resourceNames: ["kube-dns"] so the hook ServiceAccount can read the single well-known Service without broader cluster-scoped privilege. Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
Follows cozystack#2855. The talos-csr-signer image target was added on this branch before cozystack#2855 merged, so the rebase brought it in still pointing --cache-from at $(REGISTRY)/talos-csr-signer:latest, which on PR CI is the per-CI registry where :latest is never published. That cache lookup 404s on every PR build and forces a cold rebuild, exactly the failure mode cozystack#2855 fixed for the other Makefiles. Aligns the only remaining $(REGISTRY)-targeted --cache-from with the $(CACHE_REGISTRY) ghcr.io convention from cozystack#2855 so PR builds of this target warm-start off the last release's :latest. Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
db35e94 to
27dbca7
Compare
Follow-up to the CoreDNS nameserver injection. Pointing the worker's host resolver at management CoreDNS is necessary but not sufficient: kubevirt-csi calls `mount.nfs linstor-csi-nfs.cozy-linstor.svc` with no domain suffix, and management CoreDNS only knows the full FQDN (<svc>.<ns>.svc.<management-cluster-domain>). Without search domains in /etc/resolv.conf the resolver hands the partial name to CoreDNS verbatim and gets NXDOMAIN — observed in CI as the NFS mount step failing roughly 20 minutes into the E2E run. Ubuntu+kubeadm workers in main get these search domains for free from cloud-init/DHCP in the management pod network. Talos reads only what is in machineconfig, so this commit mirrors that. The chart sources management cluster domain from .Values._cluster["cluster-domain"] (populated by cozystack-controller from the platform config; default cozy.local). cluster.local is appended as a fallback when the management domain differs, covering sandbox/CI deployments that still ship with the Kubernetes default. Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
Aleksei Sviridkin (lexfrei)
left a comment
There was a problem hiding this comment.
NOT LGTM — the architecture is sound and well-documented inline, but there are functional regressions (GPU node groups, air-gapped registry mirrors), two defects in the platform migrations, and a shipped-artifact mismatch that need fixing before merge.
Business context: Phase 1 of the Kubernetes-app split (cozystack/community#8) — replace the Ubuntu+kubeadm tenant worker bootstrap with Talos via CABPT and a talos-csr-signer sidecar, rolling existing tenants over automatically.
Blockers
B1: Migrations 44/45 strip the deletion-protection label from cozystack-version
File: packages/core/platform/images/migrations/migrations/44:104-106, 45:110-112
Issue: Both migrations stamp the version via bare kubectl create configmap ... --dry-run=client -o yaml | kubectl apply -f -, which emits no labels.
Evidence: Migration 43 carries the labeled manifest inline with an explicit comment that a label-less apply by the same field manager strips the platform.cozystack.io/no-delete label; templates/cozystack-version.yaml only re-renders the labeled ConfigMap on first install. The client-side three-way merge removes the label, dropping the version anchor out of the cozystack-no-delete-guardrail ValidatingAdmissionPolicy.
Impact: After the first upgrade through these migrations, cozystack-version is silently unprotected against deletion. E2E does not exercise this path (fresh installs skip migrations).
Fix: Mirror migration 43's labeled heredoc manifest in both stamps.
B2: Migration 44 stamps the wrong version and its retry design is defeated by migration 45
File: packages/core/platform/images/migrations/migrations/44:105
Issue: The file is named 44 (runs when current version is 44) but stamps version=44 instead of 45; its failure-path comment says "leaving CURRENT_VERSION at 43". Worse: on partial failure migration 44 exits 0 without stamping, but run-migrations.sh then continues to migration 45, which on success stamps version=46 — so migration 44 never retries, defeating its own best-effort design.
Evidence: run-migrations.sh:34-46 iterates seq CURRENT TARGET-1 unconditionally on exit 0; migration 45 unconditionally stamps 46 on its own success.
Impact: If any kubectl annotate in 44 fails, the keep-annotation pass is silently skipped forever, and Helm may prune KubeadmConfigTemplates still referenced by pre-rollover MachineSets — the exact hazard migration 44 exists to prevent.
Fix: Stamp 45 in migration 44, fix the comments, and make a partial failure actually stop the chain (or otherwise guarantee a retry) so the best-effort semantics hold.
B3: GPU node groups lose the gpu=on node label — GPUs stop being exposed on Talos workers
File: packages/apps/kubernetes/templates/talos/talos-reconcile-job.yaml (worker machineconfig)
Issue: The old KubeadmConfigTemplate set node-labels: "gpu=on" for groups with gpus; the Talos machineconfig has no machine.nodeLabels at all.
Evidence: main's cluster.yaml:482-489 (label + rationale comment); the HAMi device-plugin DaemonSet selects nodes via nvidiaNodeSelector: gpu: "on" (packages/system/hami/charts/hami/values.yaml:352-353, not overridden by the in-tenant HelmRelease). The pinning test tests/gpu_node_labels_test.yaml was deleted in this PR rather than ported.
Impact: On Talos workers the HAMi device plugin stays at DESIRED=0 and no nvidia.com/* resources are advertised — a functional regression for every GPU tenant.
Fix: Add machine.nodeLabels: {gpu: "on"} for groups with gpus and restore the test.
B4: Air-gapped/registry-mirror support for tenant workers silently regresses
Files: packages/apps/kubernetes/templates/cluster.yaml:101-102 (DataVolume URL), talos-reconcile-job.yaml (install.image), templates/copy-patch-containerd.yaml (now orphaned)
Issue: Three related drops: (1) the worker boot image is fetched by CDI over HTTPS from a hardcoded https://factory.talos.dev/... with no override value; (2) install.image: factory.talos.dev/installer/... is likewise hardcoded and the machineconfig has no machine.registries block, so it cannot be mirrored; (3) the patch-containerd Secret is still copied into the tenant namespace but its only consumer (the containerd certs.d mount in the kubeadm template) was deleted — the platform registries.mirrors feature no longer reaches tenant workers.
Evidence: Pre-PR workers booted from an OCI containerDisk (main cluster.yaml:115-116), mirrorable via containerd mirrors; the public air-gapped install guide documents tenant-worker registry mirrors via patch-containerd as a supported feature; nothing consumes the copied Secret anymore and the new machineconfig has no machine.registries. The nodeHealthCheck field docs even advertise "air-gapped mirrors of factory.talos.dev", which nothing in the chart can configure.
Impact: Tenant clusters in air-gapped environments cannot create Talos workers at all (image fetch fails), and operators using registries.mirrors lose mirrors on tenant workers without any error.
Fix: Expose the image URL (and installer ref) as values defaulting to factory.talos.dev, and plumb patch-containerd/platform registries into machine.registries.mirrors — or explicitly document the loss and remove the dead copy template.
B5: diskSize and related docs now describe removed mechanics
Files: packages/apps/kubernetes/values.yaml:84, api/apps/v1alpha1/kubernetes/types.go (DiskSize), README.md (migration-41 paragraph)
Issue: "Persistent disk size for kubelet and containerd data" — on Talos this is the system disk imaged from the factory artifact; the separate kubelet disk and its persistence semantics were removed in this PR. The README paragraph about state persisting across same-VM reboots describes the deleted layout.
Evidence: the branch drops the kubelet disk and cluster.yaml now provisions a single disk-system DataVolume.
Fix: Update field descriptions and README, regenerate the schema/RD.
B6: New code paths land without tests
Files: pkg/config/config.go (ParseHelmInstallDisableWaitAnnotation), packages/apps/kubernetes/templates/talos/*
Issue: (a) The new annotation parser has zero tests while its sibling ParseHelmInstallTimeoutAnnotation has a table test in pkg/config/config_test.go:18; the HelmInstallDisableWait plumbing in rest.go is likewise untested. (b) None of the four new talos templates has helm-unittest coverage — including two behaviors that were actual bugs fixed during this branch (MHC maxUnhealthy int-vs-percent rendering, the Talos-to-Kubernetes support-matrix fail guard) and the kubelet-reservation math duplicated between cluster.yaml and the reconcile hook, which will drift unnoticed.
Fix: Add the parser table test, MHC rendering tests, support-matrix guard tests, and minimal assertions on the talos templates.
B7: talos-csr-signer.tag contradicts the build-from-source chain
File: packages/apps/kubernetes/images/talos-csr-signer.tag
Issue: The tag pins ghcr.io/clastix/talos-csr-signer:latest@sha256:827b... (upstream), while the Dockerfile/Makefile added in this PR build the binary from a pinned commit into $(REGISTRY)/talos-csr-signer:$(IMAGE_TAG)@<digest> — the regenerated tag was never committed, so the chart ships the upstream image and the from-source supply chain is dead weight.
Evidence: packages/apps/kubernetes/Makefile:22-31 writes $(REGISTRY)/talos-csr-signer:... into the tag file; compare the cpprovider counterpart, which did get its regenerated tag (ghcr.io/cozystack/cozystack/cluster-api-control-plane-provider-kamaji:v0.19.0-cozystack.0@sha256:...).
Fix: Run the build chain and commit the regenerated tag; also fix the Dockerfile comment claiming :<commit> tagging (the Makefile tags by chart version).
B8: Stale comments after the 15m-to-20m timeout bump and the TCT move
Files: templates/_helpers.tpl:77, templates/talos/bootstrap-token-tenant-job.yaml:43-44, hack/e2e-apps/run-kubernetes.sh:111, tests/cluster_test.yaml header, tests/kubelet_reservation_test.yaml header, cluster.yaml:33-39
Issue: Multiple comments still reference the 15m install timeout (now 20m via the cozyrds annotation), the removed $talosReady lookup gate, and the removed hoisted-lookups block.
Fix: Sweep and update; these comments are load-bearing for the next person debugging hook timing.
B9: Dead code left from abandoned approaches
Files: cluster.yaml:52-54 (unused $kubeletVersion/$talosVersion/$talosSchematic), talos-reconcile-job.yaml:82-85 (leftover cert-manager.io/certificates get,patch RBAC from the dropped IP-SAN patching approach — an unnecessary grant), bootstrap-token-tenant-job.yaml:58-59 (unused TENANT_KUBECONFIG env)
Fix: Remove all three; the RBAC one also narrows the hook's privileges to what it actually uses.
Non-blocking follow-ups
- The kubeadm-format bootstrap token is created without
expiration(bootstrap-token-tenant-job.yaml:87-101) — a permanent credential where CABPK previously rotated short-lived tokens;TALOS_TOKEN/CLUSTER_SECRET/bootstrap token also sit in plaintext in the TalosConfigTemplate (a non-Secret CR). The reconcile hook already re-runs on every upgrade and is a natural rotation point. - Deduplicate the ~45 lines of kubelet-reservation math mirrored between
cluster.yamlandtalos-reconcile-job.yamlinto a shared named template. - TalosConfigTemplates of removed node groups accumulate until KCP deletion (owner-ref GC only cascades from the KamajiControlPlane); consider per-group cleanup in the hook.
capi-providers-bootstraphas noupdatetarget documenting howbootstrap-components-talos.yaml/components-talos.gzwere produced (the cabpt-manager SA edit is hand-applied to a vendored bundle); the cpprovider package has one — make it symmetric.Install/Upgrade.DisableWaitchanges what "Ready" means for the Kubernetes app HelmRelease (install success no longer implies addon readiness). The cozyrds comment explains why; worth a line in the release note so operators don't read HR Ready as "workers and addons up".
| fi | ||
|
|
||
| kubectl create configmap --namespace cozy-system cozystack-version \ | ||
| --from-literal version=44 --dry-run=client --output yaml \ |
There was a problem hiding this comment.
Two problems with this stamp (B1/B2 in the review body): (1) a bare kubectl create | kubectl apply emits no labels, so the client-side three-way merge strips platform.cozystack.io/no-delete that migration 42 set and migration 43 deliberately carries inline — the ConfigMap falls out of the no-delete guardrail VAP. (2) This file runs when the current version is 44 (migration 43 already stamps 44), so it must stamp 45, not 44; the "leaving CURRENT_VERSION at 43" comment above is off by one too. Also note the partial-failure path (exit 0 without stamping) doesn't actually retry: the runner continues to migration 45, which stamps 46 on its own success, so a failed annotate pass here is silently skipped forever.
| "cozystack.io/migration-45-failed-tenants-" >/dev/null 2>&1 || true | ||
|
|
||
| kubectl create configmap --namespace cozy-system cozystack-version \ | ||
| --from-literal version=46 --dry-run=client --output yaml \ |
There was a problem hiding this comment.
Same label issue as migration 44: this bare kubectl create | kubectl apply strips the platform.cozystack.io/no-delete label from cozystack-version. Please mirror migration 43's labeled heredoc manifest (it carries an explicit comment about exactly this trap).
| - ${RELEASE}.${NS}.svc.{{ $dnsDomain }} | ||
| ca: | ||
| crt: ${TALOS_CA_B64} | ||
| kubelet: |
There was a problem hiding this comment.
The machineconfig is missing machine.nodeLabels for GPU node groups. The old KubeadmConfigTemplate set node-labels: "gpu=on" when the group has gpus, and the HAMi device-plugin DaemonSet schedules via nvidiaNodeSelector: gpu: "on" — without the label it stays at DESIRED=0 and no GPUs are advertised on Talos workers. The pinning test (tests/gpu_node_labels_test.yaml) was deleted rather than ported; please add machine.nodeLabels: {gpu: "on"} for GPU groups and restore the test.
| ignores configdrive and never applies the machineconfig. */}} | ||
| source: | ||
| http: | ||
| url: https://factory.talos.dev/image/{{ $.Values.talos.schematicID }}/{{ $.Values.talos.version }}/openstack-amd64.raw.xz |
There was a problem hiding this comment.
Hardcoding https://factory.talos.dev/... (here and in install.image in the reconcile hook) with no override value breaks air-gapped environments: pre-PR the worker image was an OCI containerDisk and could be mirrored via containerd registry mirrors; a CDI HTTP source cannot. Combined with the now-unconsumed patch-containerd Secret (no machine.registries in the machineconfig), the documented tenant-worker registry-mirror feature silently stops working. Please expose the image URL/installer ref as values and plumb registry mirrors into machine.registries.mirrors — or document the loss explicitly and drop the dead copy template.
| // callers can leave HelmInstallDisableWait zero and let flux defaults | ||
| // apply. | ||
| func ParseHelmInstallDisableWaitAnnotation(raw string) (bool, error) { | ||
| switch raw { |
There was a problem hiding this comment.
ParseHelmInstallDisableWaitAnnotation has no tests while its sibling ParseHelmInstallTimeoutAnnotation has a table test in pkg/config/config_test.go. Please add the same table coverage (empty / true variants / false variants / garbage), plus a test for the HelmInstallDisableWait plumbing in convertApplicationToHelmRelease.
| @@ -0,0 +1 @@ | |||
| ghcr.io/clastix/talos-csr-signer:latest@sha256:827b62b5fc2859d66f06f5c1f8d2473ab7109d0600d551269d8ddb98e4a39a18 | |||
There was a problem hiding this comment.
This still pins the upstream ghcr.io/clastix/talos-csr-signer:latest@sha256:... image, while the Dockerfile/Makefile in this PR build from source into $(REGISTRY)/talos-csr-signer:$(IMAGE_TAG)@<digest> — so the chart actually ships the upstream binary and the from-source chain is dead weight. Compare the cpprovider tag, which did get regenerated (...cluster-api-control-plane-provider-kamaji:v0.19.0-cozystack.0@sha256:...). Please run the build chain and commit the regenerated tag; the Dockerfile comment about :<commit> tagging also doesn't match the Makefile (it tags by chart version).
| # backoffLimit + restartPolicy: OnFailure gives the Job up to 11 attempts at | ||
| # the tenant apiserver with exponential backoff (capped at 6 min between | ||
| # restarts). Worst case ~15 min, aligned with the chart's HR install timeout | ||
| # (release.cozystack.io/helm-install-timeout: 15m on the cozyrds entry) so |
There was a problem hiding this comment.
Stale comment: the install timeout was bumped to 20m in the cozyrds annotation, but this still says 15m (same in _helpers.tpl:77 and hack/e2e-apps/run-kubernetes.sh:111). Separately, non-blocking: the bootstrap-token Secret below is created without expiration, i.e. a permanent credential where CABPK previously rotated short-lived tokens — this hook re-runs on every upgrade and would be a natural rotation point.
…nd correct version Two defects in the migration pair flagged in PR review: (1) Both stamps used `kubectl create cm ... | kubectl apply -f -`, which emits a label-less manifest. The same field manager (kubectl-client-side- apply) then strips the platform.cozystack.io/no-delete label migration 42 carried on the cozystack-version ConfigMap, dropping it out of the cozystack-no-delete-guardrail ValidatingAdmissionPolicy. After the first upgrade through 44 or 45 the version anchor was silently unprotected against deletion. The chart's templates/cozystack-version.yaml only re-renders the labeled ConfigMap on first install, so the label has to travel inline through every stamping migration. Mirrors the labeled heredoc pattern migration 43 already uses for the same reason. (2) Migration 44 stamped version=44 instead of 45 (it runs when current version is 44 and advances to 45 — same convention as every other migration in this directory). The mismatch was masked because migration 44's partial-failure path also did `exit 0` without stamping, while migration 45 on success unconditionally stamped 46 — so a partial failure in 44 silently left the keep-annotation pass undone, and the "retry on next upgrade" comment in 44's header was untrue: run- migrations.sh continued past it to migration 45, which advanced the stamp past 44's retry window forever. Best-effort semantics are not safe for migration 44 specifically: any KubeadmConfigTemplate left un-pinned is at risk of being pruned by the kubernetes chart upgrade, which then breaks the still-extant kubeadm- backed MachineSet's bootstrap.configRef while the Talos rollover is in flight. Switched to exit 1 on partial failure so run-migrations.sh halts and the operator must investigate before the chart upgrade proceeds. Migration 45 keeps its best-effort + FAILED_TENANTS annotation pattern unchanged — bumping spec.version v1.30->v1.31 on one tenant failing does not cascade into other tenants' chart upgrades. Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
…ittest
The old KubeadmConfigTemplate set kubeletExtraArgs.node-labels="gpu=on"
for nodeGroups carrying .gpus; the Talos machineconfig the post-install
hook applies had no machine.nodeLabels at all. HAMi's hami-device-plugin
DaemonSet selects nodes via nvidiaNodeSelector: gpu: "on", so on Talos
workers the plugin stayed at DESIRED=0 and no nvidia.com/* resources
were advertised — a functional regression for every GPU tenant flagged
in PR review.
Adds machine.nodeLabels: {gpu: "on"} into the TalosConfigTemplate
emitted by templates/talos/talos-reconcile-job.yaml, gated on
$group.gpus to match main's per-group conditional. The inline comment
points the next reader at the HAMi selector so the contract is obvious.
Restores tests/gpu_node_labels_test.yaml — adapted from the deleted
KubeadmConfigTemplate-targeted version to assert on the rendered post-
install Job script via matchRegex (the TCT itself is not rendered by
helm; the hook applies it at runtime), covering both the "has gpus -->
labeled" and "no gpus --> no nodeLabels" paths.
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
…apped regression PR review flagged that `templates/copy-patch-containerd.yaml` still copies the cozy-system/patch-containerd Secret into the tenant namespace but its only consumer — the containerd certs.d mount in the deleted kubeadm template — went away with the Talos rollover. The Talos machineconfig the post-install hook applies has no machine.registries block, no installer override, and the worker boot image is fetched by CDI over HTTPS directly from factory.talos.dev. The chart-shipped copy of the patch-containerd Secret is therefore unreferenced cruft and the platform-wide registries.mirrors feature no longer reaches tenant workers at all. Phase 1 of the Talos migration is scoped to the rollover itself; restoring air-gapped/mirror support for tenant workers via machine.registries.mirrors (and matching imageBase/installerImage override knobs) is non-trivial — containerd certs.d format does not translate 1:1 to the Talos registries schema — and is deferred to a follow-up. Drops the orphan template, removes the misleading "air-gapped mirrors of factory.talos.dev" hint from the nodeStartupTimeout doc (operators reading that today would assume mirrors work — they do not), and adds an explicit Breaking Changes entry to the chart README so air-gapped operators see this regression before they upgrade rather than after a failed worker boot. Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
PR review flagged three spots where dead code accumulated as the branch iterated on the post-install hook design: (1) cluster.yaml: $kubeletVersion / $talosVersion / $talosSchematic were defined at the top of the per-loop template after the TCT was moved into talos-reconcile-job.yaml — the hook redefines them in its own context and the cluster.yaml block referenced none of them. (2) talos-reconcile-job.yaml: the cert-manager.io/certificates get + patch RBAC was added when the hook patched the Talos PKI Certificate's spec.ipAddresses to include the apiserver Service ClusterIP. That approach was dropped in favour of routing workers through the apiserver Service DNS name (with extraHostEntries injecting the ClusterIP into the worker /etc/hosts), which sidesteps the TLS race against the csr- signer sidecar entirely. The RBAC grant is now unused and the hook is now strictly Talos-scoped — narrowing it removes an unnecessary cluster-write-adjacent grant for the hook ServiceAccount. (3) bootstrap-token-tenant-job.yaml: TENANT_KUBECONFIG env var was a holdover from an earlier design that distinguished the kubectl kubeconfig from a tenant-only handle; the current script reads only $KUBECONFIG (which already points at the same path), so the second variable was unreferenced. No behavioural change — the chart renders identically and helm-unittest suite stays green. Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
… Talos rollover PR review flagged that values.yaml and the generated README/CRD still describe the pre-Talos disk layout. The kubernetes chart used to provision a separate disk-kubelet PVC for kubelet/containerd state; the Talos worker boots from a single system disk imaged from the factory artifact and the chart no longer renders disk-kubelet at all. - values.yaml: diskSize description now covers the consolidated single system disk (Talos OS image + kubelet state + containerd image cache + local-path PVCs) instead of the pre-Talos "kubelet and containerd data" wording. - values.yaml: nodeHealthCheck.nodeStartupTimeout no longer hints at air-gapped mirrors of factory.talos.dev — Phase 1 has no machine.registries.mirrors plumbing, so the hint was promising behaviour the chart does not deliver (called out separately in the README Breaking Changes section). - README.md: migration-41 paragraph rewritten so the persistence model reflects the single Talos system disk rather than the removed disk-kubelet layout. - Regenerated values.schema.json, api/apps/v1alpha1/kubernetes/types.go, and packages/system/kubernetes-rd/cozyrds/kubernetes.yaml via `make generate` so the chart, Go API, and tenant CRD all carry the same description. Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
…efile tagging PR review flagged that the Dockerfile comment claimed images are mirrored into ghcr.io/cozystack/cozystack/talos-csr-signer:<commit>, but the Makefile (image-tags macro) actually tags by chart version (KUBERNETES_PKG_TAG / IMAGE_TAG) and writes the resulting <digest> into images/talos-csr-signer.tag. The upstream commit is fingerprinted indirectly via the digest, not via the tag itself. Comment-only change in Dockerfile. The images/talos-csr-signer.tag file itself still pins the upstream ghcr.io/clastix/...:latest digest and must be regenerated by running `make image-talos-csr-signer` in packages/apps/kubernetes/ (with REGISTRY=ghcr.io/cozystack/cozystack) before release — that step needs ghcr.io/cozystack/cozystack push access, which only the release pipeline / maintainer holds, so the regeneration is left to the merge handoff rather than baked into this PR. Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
PR review flagged six load-bearing comments that no longer matched reality after the chart converged on the post-install-hook design and the cozyrds helm-install-timeout was bumped 15m -> 20m: - _helpers.tpl: wait-for-kubeconfig deadline rationale referenced the 15m install timeout. The cozyrds entry now carries 20m (release.cozystack.io/helm-install-timeout) — updated, with the source of truth (cozyrds annotation) called out so the next reader finds the canonical value. - bootstrap-token-tenant-job.yaml: backoffLimit budget rationale likewise referenced 15m; the worst-case ~15m attempt window stays, it just now sits well inside the 20m HR window instead of right at the edge. - hack/e2e-apps/run-kubernetes.sh: comment above kubectl wait referenced the 15m alignment; corrected to 20m. - tests/cluster_test.yaml header: described a "$talosReady gate" that hid the TalosConfigTemplate document, which no longer exists — the TCT is produced at runtime by the talos-reconcile post-install hook, not by cluster.yaml. Rewritten to describe the actual document order the test asserts on. - tests/kubelet_reservation_test.yaml header: same $talosReady reference + a now-stale note about "happy-path TCT assertions not ported"; updated to point at gpu_node_labels_test.yaml as the in-repo example of how to assert on the hook's command literal. - templates/cluster.yaml: a "Hoist all lookups" block described an optimisation that no longer applies — the lookups moved into the talos-reconcile hook so cluster.yaml has nothing to hoist any more. Removed. No executable code changed; helm unittest stays at 89 PASS. Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
…+ new talos templates + MHC + support-matrix
PR review flagged that new code paths landed without tests. This commit
closes the gap:
- pkg/config/config_test.go: table test for the new
ParseHelmInstallDisableWaitAnnotation parser, mirroring the sibling
ParseHelmInstallTimeoutAnnotation test that already pinned that
function's behaviour. cozystack-api reads the annotation on every
ApplicationDefinition at startup and a typo here silently flips the
parent kubernetes HR into "wait for child HRs", reproducing the
Phase 1 install deadlock the disable-wait annotation was added to
break. 11 sub-cases cover empty / true / True / TRUE / false / False
/ FALSE / mixed-case (invalid) / integer (invalid) / yes (invalid) /
garbage (invalid).
- tests/talos_templates_test.yaml: renders + key-field assertions on
the four new talos/* templates (bootstrap-token-tenant-job,
talos-pki, talos-reconcile-job, talos-secrets) so a future refactor
breaking ServiceAccount/Role/CRB wiring is caught locally before
push. Uses documentIndex constants tied to source-file order with
comments noting the layout.
- tests/mhc_rendering_test.yaml: pins the MHC maxUnhealthy int-vs-
percent rendering branch — the CAPI admission webhook rejects a
quoted plain integer ("0") so the chart strips the quote when no
percent suffix is present, and the test exercises both arms.
- tests/talos_k8s_support_matrix_test.yaml: happy-path coverage for
the $talosK8sSupportMatrix guard. Failure-path coverage is left as
a follow-up — values.schema.json's Version enum is strictly tighter
than the matrix window right now, so the fail() branch is
unreachable from the test harness until either side relaxes.
Total: 11 new Go subtests + 8 new helm-unittest tests; suite goes 87
-> 97 passing.
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
…rovider's pattern PR review flagged that capi-providers-bootstrap had no update target documenting how files/bootstrap-components-talos.yaml and files/components-talos.gz were produced — the cabpt-manager ServiceAccount rename was hand-applied to a vendored bundle and the process was undocumented, blocking reproducible bumps of CABPT_UPSTREAM_VERSION. The sibling capi-providers-cpprovider package already has a working update target; this commit mirrors that pattern so both packages have symmetric maintenance flow. - Makefile: new `update` phony target that downloads the CABPT_UPSTREAM_VERSION release artifact from siderolabs/cluster-api- bootstrap-provider-talos, applies patches/, writes the result to files/bootstrap-components-talos.yaml, and regenerates the gzip bundle. CABPT_UPSTREAM_VERSION is the single bump knob; the comment block points at the two other call-sites (BootstrapProvider spec.version + the clusterctl ConfigMap name) so the next operator bumping versions can keep them in lockstep without grepping. - patches/0001-rename-default-sa-to-cabpt-manager.diff: the cozystack-side patch that replaces upstream's default ServiceAccount references with cabpt-manager, captured as a unified diff so `patch(1)` can re-apply it deterministically on top of any future CABPT_UPSTREAM_VERSION. Without the rename, capi-operator rewrites the SA namespace (`default` is the SA's literal name in upstream and capi-operator treats that as the default SA needing rewrite) and the CABPT Deployment loses its RBAC — see PR cozystack#2610 for the original diagnosis. - files/components-talos.gz: regenerated via the new target using `gzip -n` so the bundle is byte-reproducible (no embedded mtime or filename), making future `make update` runs idempotent. Decompressed content is identical to the previous file. Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
myasnikovdaniil
left a comment
There was a problem hiding this comment.
NOT LGTM — every code finding from my earlier reviews is now resolved and verified, but the worker-bootstrap rewrite has never passed E2E: CI fails in the base install on the unrelated kube-controller-manager panic (#2863; fix #2867 is not yet on this branch's base) before the kubernetes app test runs, so the chicken-and-egg fixes are verified only by code reading — on a change that rolls every existing tenant onto Talos.
Business context: Phase 1 of the Kubernetes-app split (cozystack/community#8) — replace the Ubuntu+kubeadm worker path with Talos driven by CABPT + a talos-csr-signer sidecar, existing tenants rolling over automatically.
Prior findings — all resolved ✅
Re-verified against the current head. All 10 inline + 4 body findings from the first review, plus the three follow-up findings, are addressed:
| Finding | Status | Evidence |
|---|---|---|
| Silent destructive rollover (v1.30/v1.31 → Talos) | ✅ | Support-matrix fail() guard in cluster.yaml; migration 44 pins KubeadmConfigTemplate with resource-policy=keep (strict, exit-1-retry) so Helm can't prune them mid-rollover; migration 45 bumps v1.30→v1.31 ahead of the guard |
| Talos version/schematic hardcoded ×3 | ✅ | Hoisted to values.yaml talos.version / talos.schematicID, single source |
:latest sidecar + missing resources/securityContext |
✅ | talos-csr-signer built from source, digest-pinned, resources + securityContext added |
5×N lookup calls in the nodeGroups loop |
✅ | Hoisted out of the loop, looked up once |
| Support-matrix guard + trim enum | ✅ | Matrix matches the official Talos v1.13 docs: k8s 1.31–1.36 (verified) |
dnsDomain / pod+service CIDRs hardcoded |
✅ | tenantClusterDomain helper + $podCIDR/$serviceCIDR constants |
| bootstrap-token Job retry budget (~2.5h) | ✅ | backoffLimit: 5, internal retry loop dropped |
| Hardcoded kubectl image | ✅ | images.kubectl override (digest tag) |
| cpprovider transitional fork / version skew | ✅ | Rebuilt from upstream v0.19.0 + backport patch (PR #347), image digest-pinned, 0.17–0.19 registered in metadata |
maxUnhealthy: "0" rejected by MHC webhook |
✅ | Quotes only percentage values, otherwise renders a bare int — the admission webhook accepts the default |
| MachineDeployment references a TCT the chart never renders | ✅ in code | TCT moved to the post-install talos-reconcile hook; the CAPI MachineSet blocks on its absence and unblocks the moment the hook applies it (documented inline) |
| Hook gated behind addon readiness it must itself unblock | ✅ in code | New release.cozystack.io/helm-install-disable-wait annotation → Install/Upgrade.DisableWait, so helm-controller no longer blocks the hook on in-tenant addon readiness. Wired end-to-end: ApplicationDefinition annotation → start.go parse → rest.go apply |
The three follow-up threads (maxUnhealthy / MachineDeployment→TCT / hook-ordering) are fixed in code but still marked unresolved — please resolve them.
Blocker
B1 — The Talos worker path has no passing E2E; the deadlock fixes are unverified end-to-end
The latest E2E run fails in "Install Cozystack into sandbox" with kube-controller-manager segfaulting on all three nodes — the openapi.isExtension → IsXEmbeddedResource → TypeChecker.Check → validatingadmissionpolicystatus panic tracked in #2863 (fix #2867 is still open and not on this branch's base). This is not caused by this PR — no file here touches that VAP/openapi path, and the only server-side change (start.go) just adds the disable-wait annotation parser. But it dies in the base install, so hack/e2e-apps/run-kubernetes.sh — the only thing that exercises CABPT and Talos workers — never runs.
Impact: the chicken-and-egg fixes (DisableWait + reconcile hook) are proven only by code reading and dev-cluster screenshots, on a PR that rewrites the worker bootstrap for every existing tenant.
Fix: rebase once #2867 lands (or cherry-pick it) to get a green E2E that actually boots a Talos worker through the hook path — or attach manual E2E logs showing a tenant reaching Ready on a Talos worker.
Non-blocking follow-ups
- Commit history — please rebase the churn into clean logical commits (not one giant squash). The branch carries a dead revert pair (
feat(cpprovider): rebuild kamaji control-plane provider from upstream master+ itsRevert, superseded by the v0.19.0 rebuild), regenerate-after-rebase noise (README/CRD/deepcopy/cozyrds), abandoned-approach cleanup commits, and ~25 incrementalfix(kubernetes):commits patching code introduced earlier in the same PR. The contributing checklist asks for a branch "rebased onupstream/main(no extra commits)". Suggested logical units: CABPT provider · cpprovider rebuild+patch · Talos PKI+sidecar · worker bootstrap+reconcile hook · DisableWait API · migrations · tests · docs/cleanup — and drop the revert pair entirely. DisableWaitapplies to Install and Upgrade — the tenantkubernetesHelmRelease now reportsReadyas soon as Helm completes, never blocking on actual worker/addon readiness; the safety net shifts entirely toWorkloadMonitorand the hook Jobs' own failure modes. Worth a one-line acknowledgement that any future worker-rollout regression is invisible at the HelmRelease level by design.- Migration 45 ordering — a best-effort failed v1.30→v1.31 bump leaves that tenant on v1.30, and its
kubernetesHelmRelease then hits the support-matrixfail()guard on the next reconcile (the annotation surfaces it). Is there a guaranteed ordering that migration 45 completes before the kubernetes chart upgrade reconciles? (Minor: migration 45's top comment says "leaves the version stamp at 44" — should read 45.) - Docs — confirm the website docs get a Talos-worker update (diskSize semantics, the factory.talos.dev prerequisite, air-gapped note). The in-repo README is regenerated and fine.
|
Superseded by #2931, which landed the same change on 26 June under an identical title. Closing to keep the queue honest. |
What this PR does
Phase 1 of the Kubernetes-app split design (see cozystack/community#8): replace the Ubuntu+kubeadm worker bootstrap path with Talos, driven by
cluster-api-bootstrap-provider-talos(CABPT) and aclastix/talos-csr-signersidecar embedded in the Kamaji control-plane pod. Existing tenants keep working — old machines roll out and get replaced by Talos workers without manual intervention.Highlights:
BootstrapProvideralongside the existing kubeadm one incapi-providers-bootstrap.KamajiControlPlane.spec.network.additionalServicePorts, used to publish trustd (50001/TCP) on the apiserver service.token,clusterId,clusterSecret,bootstrapToken) using the helm lookup-and-reuse pattern.bootstrap-token-<id>Secret inside the tenantkube-systemvia a Helm post-install/upgrade Job.TalosConfigTemplate(worker machineconfig,generateType: none) and switch theMachineDeployment.bootstrapreference fromKubeadmConfigTemplatetoTalosConfigTemplate, gated on Talos secrets and the Kamaji apiserver service being ready.DataVolume(source.http.url→factory.talos.dev), expose the system disk as virtio-blk withblockSize.custom: logical=512, physical=4096so 4Ki-native block storage backends (LINSTOR/DRBD) play nicely with QEMU's O_DIRECT writes and SeaBIOS still boots, drop the separate kubelet disk (Talos lays out EPHEMERAL itself).10.243.0.0/16/10.95.0.0/16) to avoid Talos's address-overlap diagnostic against the host pod CIDR.cluster.controlPlane.endpointandcilium.k8sServiceHostto the Kamaji apiserver Service ClusterIP (looked up at render time), so bootstrap survives the chicken-and-egg moment where tenant DNS does not exist yet and the host CoreDNS does not serve the tenant zone.End-to-end on a dev cluster: a fresh tenant brings up a Talos worker, kubelet CSRs get approved (apiserver-client by Kamaji, kubelet-serving by the talos-csr-signer sidecar), Cilium initialises with hostNetwork against the apiserver ClusterIP, CoreDNS comes up, and the node transitions to
Ready.Screenshots
Release note
Summary by CodeRabbit
New Features
Updates
Tests