Skip to content

feat(kubernetes): bootstrap tenant workers with Talos via CABPT - #2610

Closed
Andrei Kvapil (kvaps) wants to merge 67 commits into
cozystack:mainfrom
kvaps:phase1-talos-migration
Closed

feat(kubernetes): bootstrap tenant workers with Talos via CABPT#2610
Andrei Kvapil (kvaps) wants to merge 67 commits into
cozystack:mainfrom
kvaps:phase1-talos-migration

Conversation

@kvaps

@kvaps Andrei Kvapil (kvaps) commented May 12, 2026

Copy link
Copy Markdown
Member

What this PR does

Phase 1 of the Kubernetes-app split design (see cozystack/community#8): replace the Ubuntu+kubeadm worker bootstrap path with Talos, driven by cluster-api-bootstrap-provider-talos (CABPT) and a clastix/talos-csr-signer sidecar embedded in the Kamaji control-plane pod. Existing tenants keep working — old machines roll out and get replaced by Talos workers without manual intervention.

Highlights:

  • Add CABPT (v0.6.12) as a second BootstrapProvider alongside the existing kubeadm one in capi-providers-bootstrap.
  • Bump the Kamaji control-plane provider with an upstream-bound patch that exposes KamajiControlPlane.spec.network.additionalServicePorts, used to publish trustd (50001/TCP) on the apiserver service.
  • Generate Talos PKI (Ed25519 CA + trustd TLS) via cert-manager and stable random Talos secrets (token, clusterId, clusterSecret, bootstrapToken) using the helm lookup-and-reuse pattern.
  • Materialise a kubeadm-format bootstrap-token-<id> Secret inside the tenant kube-system via a Helm post-install/upgrade Job.
  • Render a TalosConfigTemplate (worker machineconfig, generateType: none) and switch the MachineDeployment.bootstrap reference from KubeadmConfigTemplate to TalosConfigTemplate, gated on Talos secrets and the Kamaji apiserver service being ready.
  • Boot workers from the Talos openstack image via a CDI DataVolume (source.http.urlfactory.talos.dev), expose the system disk as virtio-blk with blockSize.custom: logical=512, physical=4096 so 4Ki-native block storage backends (LINSTOR/DRBD) play nicely with QEMU's O_DIRECT writes and SeaBIOS still boots, drop the separate kubelet disk (Talos lays out EPHEMERAL itself).
  • Pin pod/service CIDRs in the worker machineconfig to the tenant ranges (10.243.0.0/16 / 10.95.0.0/16) to avoid Talos's address-overlap diagnostic against the host pod CIDR.
  • Set cluster.controlPlane.endpoint and cilium.k8sServiceHost to the Kamaji apiserver Service ClusterIP (looked up at render time), so bootstrap survives the chicken-and-egg moment where tenant DNS does not exist yet and the host CoreDNS does not serve the tenant zone.

End-to-end on a dev cluster: a fresh tenant brings up a Talos worker, kubelet CSRs get approved (apiserver-client by Kamaji, kubelet-serving by the talos-csr-signer sidecar), Cilium initialises with hostNetwork against the apiserver ClusterIP, CoreDNS comes up, and the node transitions to Ready.

Screenshots

Release note

feat(kubernetes): bootstrap tenant workers with Talos Linux instead of Ubuntu+kubeadm. Existing clusters roll over to Talos workers automatically; new tenants come up on Talos from the start. Powered by cluster-api-bootstrap-provider-talos and the talos-csr-signer sidecar in the Kamaji control-plane pod.

Summary by CodeRabbit

  • New Features

    • Talos-based bootstrapping: PKI, secrets, tenant bootstrap Job, and TalosConfigTemplate-driven worker bootstrap.
    • Cilium now prefers apiserver Service ClusterIP when available.
  • Updates

    • Control plane: Talos trust, CSR-signer sidecar, additional service-port support, and provider version bumps.
    • VM/worker: VM bootstrap readiness, disk layout and system-disk sizing changes; chart image overrides and v1.30 removal.
  • Tests

    • Cluster tests updated; new kubelet-reservation validation suite added; one legacy test suite removed.

@coderabbitai

coderabbitai Bot commented May 12, 2026

Copy link
Copy Markdown
Contributor

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review
📝 Walkthrough

Walkthrough

Adds Talos bootstrap support across the chart and system providers: new Talos bootstrap CRDs/controller, cert-manager-based Talos PKI, secrets with lookup/reuse, tenant bootstrap Job, cluster template changes to emit conditional TalosConfigTemplate and Talos wiring, plus related image/Makefile/schema/helper and test updates.

Changes

Talos bootstrap system

Layer / File(s) Summary
Bootstrap provider registration & ConfigMaps
packages/system/capi-providers-bootstrap/templates/providers.yaml, packages/system/capi-providers-bootstrap/templates/configmaps.yaml
Adds a talos BootstrapProvider; marks kubeadm ConfigMap with bootstrap-provider: kubeadm; emits a talos-specific bootstrap ConfigMap (v0.6.12-bootstrap-talos).
CRDs, RBAC, controller, webhook
packages/system/capi-providers-bootstrap/files/bootstrap-components-talos.yaml, packages/system/capi-providers-bootstrap/files/metadata-talos.yaml
Adds TalosConfig/TalosConfigTemplate CRDs (v1alpha2→v1alpha3 conversion), namespace, RBAC, webhook, cert-manager webhook certs, controller Deployment, and provider metadata mapping Talos release series.
Talos PKI (cert-manager)
packages/apps/kubernetes/templates/talos/talos-pki.yaml
Creates self-signed bootstrap Issuer, Ed25519 CA Certificate + Secret, CA-backed Issuer, and server TLS Certificate with DNS SANs and optional ClusterIP SAN via Helm lookup.
Talos secrets with lookup+reuse
packages/apps/kubernetes/templates/talos/talos-secrets.yaml
Renders <release>-talos-secrets Secret: looks up existing Secret to reuse decoded values or generates random token, clusterId, clusterSecret, plus bootstrap token components; writes base64-encoded data keys.
Tenant bootstrap token Job
packages/apps/kubernetes/templates/talos/bootstrap-token-tenant-job.yaml
Adds Helm post-install/post-upgrade hook: ServiceAccount + Job that mounts admin-kubeconfig, runs kubectl to apply tenant kubeadm bootstrap-token Secret and namespace-scoped RBAC, deletes legacy cluster-scoped RBAC, with backoff/TTL/restartPolicy controls.
Cluster template: VM wiring & Kamaji control-plane Talos integration
packages/apps/kubernetes/templates/cluster.yaml
Hoists lookup results (apiserver Service ClusterIP and Talos secrets) and computes a $talosReady gate; switches KubeVirt virtualMachineBootstrapCheck.checkStrategy to none; replaces disk-kubeletdisk-system DataVolumeTemplate and explicit blockSize.custom; augments Kamaji control plane with talos-trustd port, apiserver ClusterIP in certSANs, and injects talos-csr-signer extra container + Secret volumes with TALOS_TOKEN wiring.
Worker Talos bootstrap template generation
packages/apps/kubernetes/templates/cluster.yaml
Replaces kubeadm worker bootstrap with a conditional bootstrap.cluster.x-k8s.io/v1alpha3 TalosConfigTemplate (generateType: none) emitted only when looked-up Secrets/ClusterIP are present; embeds Talos worker config (token/CA, kubelet image, reservations/eviction, installer disk/image, cluster endpoint/IDs/CA/credentials).
MachineDeployment bootstrap reference
packages/apps/kubernetes/templates/cluster.yaml
Updates MachineDeployment spec.template.spec.bootstrap.configRef to reference bootstrap.cluster.x-k8s.io/v1alpha3 TalosConfigTemplate instead of the prior KubeadmConfigTemplate.
Control-plane API/schema changes & backport
packages/system/capi-providers-cpprovider/files/control-plane-components.yaml, packages/system/capi-providers-cpprovider/images/.../patches/0001-backport-additional-service-ports-pr347.diff
Adds additionalServicePorts API/schema and plumbing, introduces userAnnotations and configurationJSONPatches schemas, updates multiple schema docs (taint operator docs, resizePolicy/procMount text), and updates controller image reference to a cozystack variant.

Tooling, images, schema, tests

Layer / File(s) Summary
Helm helpers & consumer defaults
packages/apps/kubernetes/templates/_helpers.tpl, packages/apps/kubernetes/templates/helmreleases/ouroboros.yaml
Adds kubernetes.tenantClusterDomain helper (returns cluster.local) and uses it for ouroboros clusterDomain default.
Cilium service host discovery
packages/apps/kubernetes/templates/helmreleases/cilium.yaml
cozystack.defaultCiliumValues now lookups the release Service spec.clusterIP for cilium.k8sServiceHost, falling back to <release>.<namespace>.svc.
Values/schema and API types updates
packages/apps/kubernetes/values.yaml, packages/apps/kubernetes/values.schema.json, api/apps/v1alpha1/kubernetes/types.go
Removes v1.30 from supported Kubernetes versions; adds images.kubectl and images.talosCsrSigner overrides and a talos config block in values; adds Talos type and new image fields to the API; tightens ResourcesPreset enum (drops v1.30).
Images, Dockerfiles, Makefiles
packages/apps/kubernetes/images/talos-csr-signer/Dockerfile, packages/apps/kubernetes/images/*.tag, packages/apps/kubernetes/Makefile, Makefile, packages/system/capi-providers-cpprovider/Makefile, packages/system/capi-providers-cpprovider/images/.../Dockerfile
Adds talos-csr-signer Dockerfile and build target; updates kubectl and talos-csr-signer tag files; removes ubuntu-container-disk Dockerfile and pinned container-disk tag entries; updates package/root Makefiles to build kamaji provider and talos-csr-signer images and persist digested tags; adds kamaji provider Dockerfile and Makefile automation (build/update with patches).
Tests: removed and added suites; test index updates
packages/apps/kubernetes/tests/kubelet-reservation_test.yaml (removed), packages/apps/kubernetes/tests/kubelet_reservation_test.yaml (added), packages/apps/kubernetes/tests/cluster_test.yaml
Removes old kubelet-reservation test; adds new kubelet_reservation_test.yaml with extensive validation-failure cases for kubelet reservation/eviction logic; updates cluster_test.yaml indices and replaces disk-kubelet assertions with disk-system checks; removes some KubeadmConfigTemplate-related test blocks.
Misc: versions and docs
packages/apps/kubernetes/files/versions.yaml, packages/apps/kubernetes/README.md, packages/system/kubernetes-rd/cozyrds/kubernetes.yaml
Removes v1.30 mapping from versions.yaml; documents images.kubectl and images.talosCsrSigner and talos parameters in README; updates ApplicationDefinition OpenAPI Chart Values schema and keysOrder to include new fields and drop v1.30.

sequenceDiagram
mermaid
sequenceDiagram
participant Helm as Helm
participant CertMgr as Cert-Manager
participant CAPI as CAPI Bootstrap Controller
participant Tenant as Tenant API Server
participant VM as Worker VM (Talos)
participant ControlPlane as Kamaji ControlPlane

Helm->>CertMgr: render talos-pki (create Issuer/CA/TLS cert)
Helm->>Helm: render talos-secrets (lookup or generate)
Helm->>CAPI: emit TalosConfigTemplate referencing secrets + apiserver SAN
CAPI->>Tenant: Job applies kubeadm bootstrap-token Secret in tenant kube-system
VM->>CAPI: request Talos config/discovery
CAPI->>VM: provide TalosConfigTemplate data
VM->>ControlPlane: talos trustd CSR -> talos-csr-signer sidecar
ControlPlane->>VM: signed cert returned; node registers

Estimated Code Review Effort

🎯 4 (Complex) | ⏱️ ~60 minutes

Suggested reviewers

  • lexfrei
  • myasnikovdaniil
  • IvanHunters
  • lllamnyp

Poem

🐰 I stitched a secret, soft and neat,
Tokens nested where talos and k8s meet,
Certs that bloom from cert-manager beds,
Jobs that plant tokens so nodes find their threads,
Little signer hops — and the cluster sleeps complete.

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly and specifically summarizes the main change: introducing Talos-based worker bootstrap for tenant Kubernetes clusters via CABPT (Cluster API Bootstrap Provider for Talos), which is the core objective of this feature PR.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@github-actions github-actions Bot added size/XXL This PR changes 1000+ lines, ignoring generated files area/kubernetes Issues or PRs related to the tenant Kubernetes app kind/feature Categorizes issue or PR as related to a new feature labels May 12, 2026
@kvaps
Andrei Kvapil (kvaps) marked this pull request as ready for review May 12, 2026 10:50

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

@gemini-code-assist

Copy link
Copy Markdown
Contributor

Summary of Changes

Hello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request implements Phase 1 of the Kubernetes-app split design by transitioning tenant worker node bootstrapping from Ubuntu+kubeadm to Talos Linux. The changes include integrating the Talos bootstrap provider, enhancing the Kamaji control-plane to support necessary sidecar services, and automating the lifecycle of required PKI and cluster secrets. These updates ensure that existing tenants can transition to Talos workers automatically, while new tenants will utilize Talos by default, improving overall system robustness and alignment with modern cloud-native infrastructure practices.

Highlights

  • Talos Bootstrap Integration: Replaced the Ubuntu+kubeadm worker bootstrap path with Talos Linux, utilizing cluster-api-bootstrap-provider-talos (CABPT) and a talos-csr-signer sidecar.
  • Kamaji Control Plane Update: Updated the Kamaji control-plane provider to expose additional service ports, enabling the publication of the trustd service on the apiserver.
  • Automated PKI and Secret Management: Implemented automated generation of Talos PKI via cert-manager and managed stable random secrets using a Helm lookup-and-reuse pattern.
  • Optimized Disk Configuration: Configured Talos worker disks with virtio-blk and custom block sizes (512/4096) to ensure compatibility with 4Ki-native storage backends like LINSTOR.
New Features

🧠 You can now enable Memory (public preview) to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature Command Description
Code Review /gemini review Performs a code review for the current pull request in its current state.
Pull Request Summary /gemini summary Provides a summary of the current pull request in its current state.
Comment Gemini (@gemini-code-assist) Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help /gemini help Displays a list of available commands.

Customization

To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on Gemini (@gemini-code-assist) comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

Footnotes

  1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution.

@dosubot dosubot Bot added the area/platform Issues or PRs related to platform infrastructure (bundle, flux, talos, installer) label May 12, 2026

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 3

🧹 Nitpick comments (4)
packages/system/capi-providers-cpprovider/files/control-plane-components.yaml (3)

6686-6718: 💤 Low value

Consider adding operation-specific validation rules.

The configurationJSONPatches schema allows any combination of fields for all operations, but RFC 6902 requires specific fields for certain operations:

  • from is required for move and copy operations
  • value is required for add, replace, and test operations but not valid for remove

While the current design provides flexibility, adding CEL validation rules could catch misconfigurations at admission time rather than at runtime.

Example CEL validation rules
x-kubernetes-validations:
- message: "from is required for move and copy operations"
  rule: "!(self.op == 'move' || self.op == 'copy') || has(self.from)"
- message: "value is required for add, replace, and test operations"  
  rule: "!(self.op == 'add' || self.op == 'replace' || self.op == 'test') || has(self.value)"

Also applies to: 13836-13868

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In
`@packages/system/capi-providers-cpprovider/files/control-plane-components.yaml`
around lines 6686 - 6718, Add CEL admission validations to the
configurationJSONPatches schema to enforce RFC 6902 operation-specific required
fields: ensure that when configurationJSONPatches.items.properties.op equals
"move" or "copy" the "from" field must be present, and when op equals "add",
"replace", or "test" the "value" field must be present (and optionally forbid
"value" for "remove"); implement these as x-kubernetes-validations entries on
configurationJSONPatches.items (referencing op, from, and value) so
misconfigured patches are rejected at admission time.

170-14491: Clarify CRD maintenance and generation process.

This CRD file contains significant schema changes (additionalServicePorts, userAnnotations, configurationJSONPatches) that align with the custom controller image version v0.16.0-talos-csr-signer.0.

Since this appears to be a custom fork of the Kamaji control-plane provider, please confirm:

  1. Is this CRD manually maintained or generated from the custom fork?
  2. Is there a process to keep it synchronized with upstream Kamaji updates?
  3. Should there be a comment or annotation documenting the customizations?
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In
`@packages/system/capi-providers-cpprovider/files/control-plane-components.yaml`
around lines 170 - 14491, The CRD file contains extensive schema changes related
to features like additionalServicePorts, userAnnotations, and
configurationJSONPatches and matches the custom controller image version
v0.16.0-talos-csr-signer.0. Please clarify if this CRD is manually maintained or
automatically generated from the custom Kamaji fork, and describe the process
used to keep it in sync with upstream Kamaji updates. Also, add a clear comment
or annotation in the CRD file documenting that this is a customized version and
specify the related controller image and version to help future maintainers
understand the source and divergence from upstream.

5896-5913: ⚡ Quick win

Add schema-level pattern validation for domain-prefixed keys in userAnnotations.

The userAnnotations schema documents that "all keys must be domain-prefixed" but doesn't enforce this requirement via pattern validation. Users could specify non-prefixed keys that pass admission but be rejected later by the signer or cause unexpected behavior. Adding a pattern validation rule on additionalProperties would catch these configuration errors at admission time.

Applies to both occurrences (lines 5896-5913 and 13046-13063).

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In
`@packages/system/capi-providers-cpprovider/files/control-plane-components.yaml`
around lines 5896 - 5913, Add schema-level validation to enforce domain-prefixed
keys for the userAnnotations object: update the userAnnotations schema (the
object with additionalProperties) to include a propertyNames or
patternProperties rule that requires keys to match a domain-prefixed pattern
(e.g. a DNS subdomain followed by a slash and a name) so invalid non-prefixed
keys are rejected at admission; apply the same change to both occurrences of
userAnnotations (the block describing spec.unverifiedUserAnnotations) and ensure
the pattern used is consistent with Kubernetes DNS subdomain rules (for example:
DNS-subdomain '/' name).
packages/apps/kubernetes/templates/cluster.yaml (1)

50-50: ⚡ Quick win

Consider centralizing Talos version and schematic ID.

The Talos version (v1.13.0) and schematic ID (ce4c980...) are hardcoded in three locations: the dataVolume source URL (line 50), template variables (lines 510-511), and the machineconfig install image (line 556). If these values drift, workers would boot from one image but attempt to install from another, causing failures.

♻️ Refactor to define once at template scope
 {{- if not $etcd }}
 ---
+{{- $talosVersion := "v1.13.0" }}
+{{- $talosSchematic := "ce4c980550dd2ab1b17bbf2b08801c7eb59418eafe8f279833297925d67c7515" }}
 apiVersion: v1
 kind: ConfigMap

Then reference {{ $talosVersion }} and {{ $talosSchematic }} at lines 50, 510-511, and 556, or move these to values.yaml for user configurability.

Also applies to: 510-511, 556-556

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/apps/kubernetes/templates/cluster.yaml` at line 50, The Talos
schematic ID and version are hardcoded in multiple places (the dataVolume source
URL string, the template variable block around the current schematic/version,
and the machineconfig install image), so define single template-scoped variables
(e.g., $talosSchematic and $talosVersion or put them in values.yaml) and replace
the literal parts in the dataVolume URL (line with url), the template variable
declarations (the block currently holding the schematic/version), and the
machineconfig install image reference to use those variables (e.g., {{
$talosSchematic }} and {{ $talosVersion }}) so all three locations share the
same source of truth.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@packages/apps/kubernetes/templates/talos/bootstrap-token-tenant-job.yaml`:
- Line 49: The kubectl image is hardcoded to alpine/k8s:1.33.4 in the
bootstrap-token-tenant Job; update it to be parameterized or documented: change
the image reference in the template for the Job (where image:
docker.io/alpine/k8s:1.33.4 is set) to use the chart value (e.g. values.version)
like other templates, or if 1.33.4 is intentionally required, add a clear
comment above that image line explaining the backward-compatibility rationale
and why the 2-minor-version skew is acceptable.

In
`@packages/system/capi-providers-cpprovider/files/control-plane-components.yaml`:
- Around line 6744-6795: The schema for additionalServicePorts lacks validation
for reserved port names and numeric port ranges: update the properties under
additionalServicePorts to add an x-kubernetes-validations rule on the name
property (e.g., disallow values 'kube-apiserver' and 'konnectivity-server'), add
minimum: 1 and maximum: 65535 to the port property, and modify targetPort's
integer anyOf branch to include minimum: 1 and maximum: 65535 while keeping the
string branch and x-kubernetes-int-or-string flag; apply the same changes to the
corresponding block referenced at the other location.
- Line 14491: The image tag in control-plane-components.yaml currently uses
ghcr.io/cozystack/cozystack/cluster-api-control-plane-provider-kamaji:v0.16.0-talos-csr-signer.0;
update this image value to use the same tag used elsewhere in the PR
(v0.16.0-cp-talos-csr-signer.1) so it matches configmaps.yaml and
providers.yaml, ensuring the image field for the
cluster-api-control-plane-provider-kamaji entry is replaced with the
v0.16.0-cp-talos-csr-signer.1 tag.

---

Nitpick comments:
In `@packages/apps/kubernetes/templates/cluster.yaml`:
- Line 50: The Talos schematic ID and version are hardcoded in multiple places
(the dataVolume source URL string, the template variable block around the
current schematic/version, and the machineconfig install image), so define
single template-scoped variables (e.g., $talosSchematic and $talosVersion or put
them in values.yaml) and replace the literal parts in the dataVolume URL (line
with url), the template variable declarations (the block currently holding the
schematic/version), and the machineconfig install image reference to use those
variables (e.g., {{ $talosSchematic }} and {{ $talosVersion }}) so all three
locations share the same source of truth.

In
`@packages/system/capi-providers-cpprovider/files/control-plane-components.yaml`:
- Around line 6686-6718: Add CEL admission validations to the
configurationJSONPatches schema to enforce RFC 6902 operation-specific required
fields: ensure that when configurationJSONPatches.items.properties.op equals
"move" or "copy" the "from" field must be present, and when op equals "add",
"replace", or "test" the "value" field must be present (and optionally forbid
"value" for "remove"); implement these as x-kubernetes-validations entries on
configurationJSONPatches.items (referencing op, from, and value) so
misconfigured patches are rejected at admission time.
- Around line 170-14491: The CRD file contains extensive schema changes related
to features like additionalServicePorts, userAnnotations, and
configurationJSONPatches and matches the custom controller image version
v0.16.0-talos-csr-signer.0. Please clarify if this CRD is manually maintained or
automatically generated from the custom Kamaji fork, and describe the process
used to keep it in sync with upstream Kamaji updates. Also, add a clear comment
or annotation in the CRD file documenting that this is a customized version and
specify the related controller image and version to help future maintainers
understand the source and divergence from upstream.
- Around line 5896-5913: Add schema-level validation to enforce domain-prefixed
keys for the userAnnotations object: update the userAnnotations schema (the
object with additionalProperties) to include a propertyNames or
patternProperties rule that requires keys to match a domain-prefixed pattern
(e.g. a DNS subdomain followed by a slash and a name) so invalid non-prefixed
keys are rejected at admission; apply the same change to both occurrences of
userAnnotations (the block describing spec.unverifiedUserAnnotations) and ensure
the pattern used is consistent with Kubernetes DNS subdomain rules (for example:
DNS-subdomain '/' name).
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: bd1a45cd-4241-413d-9d86-c6c48dc95857

📥 Commits

Reviewing files that changed from the base of the PR and between 86f5267 and 8ad1b31.

⛔ Files ignored due to path filters (2)
  • packages/system/capi-providers-bootstrap/files/components-talos.gz is excluded by !**/*.gz
  • packages/system/capi-providers-cpprovider/files/components.gz is excluded by !**/*.gz
📒 Files selected for processing (14)
  • packages/apps/kubernetes/templates/cluster.yaml
  • packages/apps/kubernetes/templates/helmreleases/cilium.yaml
  • packages/apps/kubernetes/templates/talos/bootstrap-token-tenant-job.yaml
  • packages/apps/kubernetes/templates/talos/talos-pki.yaml
  • packages/apps/kubernetes/templates/talos/talos-secrets.yaml
  • packages/apps/kubernetes/tests/cluster_test.yaml
  • packages/apps/kubernetes/tests/kubelet-reservation_test.yaml
  • packages/system/capi-providers-bootstrap/files/bootstrap-components-talos.yaml
  • packages/system/capi-providers-bootstrap/files/metadata-talos.yaml
  • packages/system/capi-providers-bootstrap/templates/configmaps.yaml
  • packages/system/capi-providers-bootstrap/templates/providers.yaml
  • packages/system/capi-providers-cpprovider/files/control-plane-components.yaml
  • packages/system/capi-providers-cpprovider/templates/configmaps.yaml
  • packages/system/capi-providers-cpprovider/templates/providers.yaml
💤 Files with no reviewable changes (1)
  • packages/apps/kubernetes/tests/kubelet-reservation_test.yaml

Comment thread packages/apps/kubernetes/templates/talos/bootstrap-token-tenant-job.yaml Outdated
Comment thread packages/system/capi-providers-cpprovider/files/control-plane-components.yaml Outdated

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request transitions the Kubernetes package from Kubeadm to Talos Linux bootstrapping and updates the Cluster API providers. It introduces TalosConfigTemplate, sidecar-based CSR signing, and automated PKI management. The reviewer feedback highlights several critical improvements: pinning the talos-csr-signer image version instead of using :latest, adding mandatory resource requests, limits, and security contexts for new containers, and externalizing hardcoded configurations—such as subnets, DNS domains, and Talos image metadata—into values.yaml to ensure the templates are configurable and maintainable.

Comment thread packages/apps/kubernetes/templates/cluster.yaml Outdated
Comment thread packages/apps/kubernetes/templates/cluster.yaml Outdated
Comment thread packages/apps/kubernetes/templates/cluster.yaml
Comment thread packages/apps/kubernetes/templates/cluster.yaml
Comment thread packages/apps/kubernetes/templates/cluster.yaml Outdated
Comment thread packages/apps/kubernetes/templates/cluster.yaml Outdated
Comment thread packages/apps/kubernetes/templates/cluster.yaml Outdated
Comment thread packages/apps/kubernetes/templates/talos/bootstrap-token-tenant-job.yaml Outdated

@myasnikovdaniil myasnikovdaniil left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for tackling this — the architectural direction is solid and the rationale comments throughout (checkStrategy: none, openstack-vs-nocloud, lookup-and-reuse, hook ordering, namespace-scoped endpoint-discovery RBAC) make the PR pleasant to read. Asking for changes on a few concrete points before merge:

Critical — destructive upgrade hazard for existing tenants

The Talos worker image is hardcoded at v1.13.0, which officially supports K8s v1.32–v1.34 only. The chart still advertises v1.30–v1.35 in the Version enum (values.yaml) and files/versions.yaml is unchanged. On helm upgrade of an existing tenant running version: v1.30 (or v1.31), the KubevirtMachineTemplate hash rolls, MachineDeployment rolls workers to TalosConfigTemplate, and new Talos workers come up running kubelet:v1.30.14 under Talos v1.13 — an unsupported combination, with no operator-facing warning. HelmRelease succeeds, customer workloads end up on a broken substrate.

This must be a chart-level hard fail, not just a release-note. Suggested guard at the top of cluster.yaml (where $talosVersion is set):

{{- $talosK8sSupportMatrix := dict
      "v1.13.0" (list "v1.32" "v1.33" "v1.34")
}}
{{- $supported := index $talosK8sSupportMatrix $talosVersion }}
{{- if not (has .Values.version $supported) }}
{{-   fail (printf "Kubernetes %s is not supported by Talos %s. Bump tenant 'version' to one of: %v." .Values.version $talosVersion $supported) }}
{{- end }}

Failed HelmRelease is visible in Flux/dashboards; silent rollover is not. Alternatively, consider gating this PR's Talos path behind an opt-in bootstrapMode: kubeadm|talos flag so existing tenants on v1.30/v1.31 migrate forward at their own pace. Either way, also trim the Version enum + files/versions.yaml to the Talos-supported set.

Process — 10 unanswered bot inline comments

CodeRabbit (3 actionable + 4 nitpicks) and Gemini Code Assist (7 inline) have all 10 threads with in_reply_to_id: null — zero engagement. Several findings overlap with mine below (:latest, hardcoded Talos version, hardcoded CIDRs, hardcoded dnsDomain, missing resources/securityContext on the csr-signer sidecar). Please address each thread with a fix or a brief "won't fix because X" so it's clear which findings are deliberate trade-offs.

Test coverage

tests/kubelet-reservation_test.yaml (1843 lines) was deleted, but the underlying reservation/validation logic in cluster.yaml (5%-of-memory auto-compute, 256Mi floor, 1Gi ceiling, format guards, mixed-unit-type rejection, total-reserved < effective-memory invariant) is still live and now feeds the new TalosConfigTemplate.spec.template.spec.data. Please port the assertions to match the new TCT output, or at minimum retain the failure-path tests (those don't depend on the lookup gate and would still render).

Cleanup

packages/apps/kubernetes/images/ubuntu-container-disk-v1.{30,31,32,33,34,35}.tag (6 files) are now dead — the containerDisk.image reference is gone. Delete them as part of this PR.

Inline comments below cover the per-line items (:latest image, hardcoded Talos coords, CIDRs/dnsDomain, dedup of lookups, version skew, Job retry-budget, etc.). Most of these mirror what CodeRabbit and Gemini already flagged.

Recommendation: request changes.

Comment thread packages/apps/kubernetes/templates/cluster.yaml Outdated
Comment thread packages/apps/kubernetes/templates/cluster.yaml Outdated
Comment thread packages/apps/kubernetes/templates/cluster.yaml Outdated
Comment thread packages/apps/kubernetes/templates/cluster.yaml Outdated
Comment thread packages/apps/kubernetes/templates/cluster.yaml Outdated
Comment thread packages/apps/kubernetes/templates/cluster.yaml Outdated
Comment thread packages/apps/kubernetes/templates/cluster.yaml Outdated
Comment thread packages/apps/kubernetes/templates/talos/bootstrap-token-tenant-job.yaml Outdated
Comment thread packages/apps/kubernetes/templates/talos/bootstrap-token-tenant-job.yaml Outdated
Comment thread packages/system/capi-providers-cpprovider/files/control-plane-components.yaml Outdated
@kvaps

Copy link
Copy Markdown
Member Author

The Talos worker image is hardcoded at v1.13.0, which officially supports K8s v1.32–v1.34 only.

Per the official Talos support matrix, Talos v1.13 supports Kubernetes 1.31, 1.32, 1.33, 1.34, 1.35 and 1.36 — not just 1.32–1.34: https://docs.siderolabs.com/talos/v1.13/getting-started/support-matrix

So the only entry in the chart's Version enum that v1.13 does not cover is v1.30 (for 1.30 the minimum is Talos v1.11). I agree the silent rollover is still a real hazard for tenants on version: v1.30 — I'll add a chart-level hard fail and trim v1.30 from Version / files/versions.yaml (or gate it behind an opt-in Talos image override for the v1.11/1.12 line if we want to keep 1.30 reachable).

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 4

♻️ Duplicate comments (1)
packages/apps/kubernetes/images/talos-csr-signer/Dockerfile (1)

8-8: ⚠️ Potential issue | 🟠 Major

Verify the golang base image version.

The Dockerfile specifies golang:1.26, but Go 1.26 does not exist yet (latest is approximately 1.23 as of March 2025).

See the identical issue in packages/system/capi-providers-cpprovider/images/cluster-api-control-plane-provider-kamaji/Dockerfile line 8. Both Dockerfiles should use the same valid Go version.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/apps/kubernetes/images/talos-csr-signer/Dockerfile` at line 8, The
Dockerfile's base image line "FROM golang:1.26" uses a non-existent Go version;
update that line to a valid released Go image (e.g., "FROM golang:1.23" or the
project-standard version) and make the identical change in the other Dockerfile
that currently uses "FROM golang:1.26" so both images use the same, supported Go
version.
🧹 Nitpick comments (2)
packages/apps/kubernetes/templates/helmreleases/ouroboros.yaml (1)

22-43: 💤 Low value

Consider updating the comment to reflect the helper function usage.

The comment on line 22 states "Pin clusterDomain to cluster.local" but the implementation now uses a helper function. While the comment remains accurate in spirit (the helper returns cluster.local by default), it could be slightly more precise by mentioning the helper, for example:

Pin clusterDomain via the kubernetes.tenantClusterDomain helper (defaults to cluster.local).

This is optional and does not affect functionality; the existing comment still correctly guides operators to the override path in lines 40-43.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/apps/kubernetes/templates/helmreleases/ouroboros.yaml` around lines
22 - 43, Update the top comment to state that clusterDomain is pinned via the
kubernetes.tenantClusterDomain helper (which defaults to "cluster.local")
instead of asserting a hardcoded value; e.g. replace "Pin clusterDomain to
cluster.local" with "Pin clusterDomain via the kubernetes.tenantClusterDomain
helper (defaults to cluster.local)". Keep the existing guidance about overriding
via addons.ouroboros.valuesOverride.ouroboros.controller.clusterDomain and
preserve the explanatory context about tenant CoreDNS and kubelet behavior.
packages/system/capi-providers-cpprovider/images/cluster-api-control-plane-provider-kamaji/Dockerfile (1)

19-20: ⚖️ Poor tradeoff

Consider adding checksum verification for the downloaded source tarball.

The Dockerfile downloads upstream source from GitHub without verifying its integrity. An attacker who compromises the upstream repository or performs a man-in-the-middle attack could inject malicious code into the build.

Consider pinning a checksum (SHA256) alongside the COMMIT ARG and verifying it:

ARG COMMIT=e52ee00a9f5fcb516314ab04da0c668a3dfaca54
ARG COMMIT_SHA256=<expected_sha256_of_tarball>

RUN curl -sSL "https://github.com/clastix/cluster-api-control-plane-provider-kamaji/archive/${COMMIT}.tar.gz" \
    -o source.tar.gz && \
    echo "${COMMIT_SHA256}  source.tar.gz" | sha256sum -c - && \
    tar -xzf source.tar.gz --strip=1 && \
    rm source.tar.gz

This adds supply-chain security at the cost of maintaining the checksum when bumping COMMIT.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In
`@packages/system/capi-providers-cpprovider/images/cluster-api-control-plane-provider-kamaji/Dockerfile`
around lines 19 - 20, Add checksum verification when downloading the upstream
tarball in the Dockerfile: introduce an ARG for the expected SHA256 (e.g.,
COMMIT_SHA256) alongside COMMIT, download the tarball to a temporary file
instead of streaming, verify it with sha256sum -c using the COMMIT_SHA256, only
then extract with tar --strip=1 and remove the temp file; update the RUN that
references COMMIT to perform these steps and fail the build if the checksum
check fails.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@packages/apps/kubernetes/images/talos-csr-signer/Dockerfile`:
- Around line 13-14: The Dockerfile declares build args TARGETOS and TARGETARCH
without defaults causing builds to fail when not supplied; update the ARG
declarations (both the first occurrences and the later ones around lines showing
21-22) to provide sensible defaults (e.g., set TARGETOS=linux and
TARGETARCH=amd64) so go build produces a correct binary when build args are
omitted; ensure you change ARG TARGETOS and ARG TARGETARCH to include the
default values and keep the same argument names used in the go build invocation.

In `@packages/apps/kubernetes/templates/cluster.yaml`:
- Line 644: Inside the nodeGroups range the dot (.) is the nodeGroup object, so
calling include "kubernetes.tenantClusterDomain" . passes the wrong context;
change the include to use the root context (e.g., include
"kubernetes.tenantClusterDomain" $) so the tenantClusterDomain template receives
the chart/global root context instead of the nodeGroup object.

In `@packages/apps/kubernetes/templates/talos/bootstrap-token-tenant-job.yaml`:
- Around line 55-58: The KUBECONFIG/TENANT_KUBECONFIG env vars point to
/tenant-admin/admin.svc but the secret key used by the control-plane is
super-admin.svc; update the environment values so both TENANT_KUBECONFIG and
KUBECONFIG reference /tenant-admin/super-admin.svc (or change to the actual
secret key name used by the control-plane) so kubectl can load the correct
kubeconfig when the job (env names TENANT_KUBECONFIG and KUBECONFIG) runs.

In
`@packages/system/capi-providers-cpprovider/images/cluster-api-control-plane-provider-kamaji/Dockerfile`:
- Around line 14-15: The Dockerfile declares ARG TARGETOS and ARG TARGETARCH but
only TARGETOS has a fallback; make defaults consistent by declaring ARG
TARGETOS=linux and ARG TARGETARCH=amd64 (or another desired arch) so builds work
when variables are unset, and then update the build invocation (the RUN that
calls go build) to reference ${TARGETOS} and ${TARGETARCH} directly (removing
inline shell fallbacks) so the ARG defaults are authoritative; locate the ARG
lines and the RUN go build step in the Dockerfile to apply the change.

---

Duplicate comments:
In `@packages/apps/kubernetes/images/talos-csr-signer/Dockerfile`:
- Line 8: The Dockerfile's base image line "FROM golang:1.26" uses a
non-existent Go version; update that line to a valid released Go image (e.g.,
"FROM golang:1.23" or the project-standard version) and make the identical
change in the other Dockerfile that currently uses "FROM golang:1.26" so both
images use the same, supported Go version.

---

Nitpick comments:
In `@packages/apps/kubernetes/templates/helmreleases/ouroboros.yaml`:
- Around line 22-43: Update the top comment to state that clusterDomain is
pinned via the kubernetes.tenantClusterDomain helper (which defaults to
"cluster.local") instead of asserting a hardcoded value; e.g. replace "Pin
clusterDomain to cluster.local" with "Pin clusterDomain via the
kubernetes.tenantClusterDomain helper (defaults to cluster.local)". Keep the
existing guidance about overriding via
addons.ouroboros.valuesOverride.ouroboros.controller.clusterDomain and preserve
the explanatory context about tenant CoreDNS and kubelet behavior.

In
`@packages/system/capi-providers-cpprovider/images/cluster-api-control-plane-provider-kamaji/Dockerfile`:
- Around line 19-20: Add checksum verification when downloading the upstream
tarball in the Dockerfile: introduce an ARG for the expected SHA256 (e.g.,
COMMIT_SHA256) alongside COMMIT, download the tarball to a temporary file
instead of streaming, verify it with sha256sum -c using the COMMIT_SHA256, only
then extract with tar --strip=1 and remove the temp file; update the RUN that
references COMMIT to perform these steps and fail the build if the checksum
check fails.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 7640461c-29ab-4337-ae7c-056bcf5a08c1

📥 Commits

Reviewing files that changed from the base of the PR and between 8ad1b31 and 630c952.

⛔ Files ignored due to path filters (1)
  • packages/system/capi-providers-cpprovider/files/components.gz is excluded by !**/*.gz
📒 Files selected for processing (29)
  • Makefile
  • api/apps/v1alpha1/kubernetes/types.go
  • packages/apps/kubernetes/Makefile
  • packages/apps/kubernetes/README.md
  • packages/apps/kubernetes/files/versions.yaml
  • packages/apps/kubernetes/images/kubectl.tag
  • packages/apps/kubernetes/images/talos-csr-signer.tag
  • packages/apps/kubernetes/images/talos-csr-signer/Dockerfile
  • packages/apps/kubernetes/images/ubuntu-container-disk-v1.30.tag
  • packages/apps/kubernetes/images/ubuntu-container-disk-v1.31.tag
  • packages/apps/kubernetes/images/ubuntu-container-disk-v1.32.tag
  • packages/apps/kubernetes/images/ubuntu-container-disk-v1.33.tag
  • packages/apps/kubernetes/images/ubuntu-container-disk-v1.34.tag
  • packages/apps/kubernetes/images/ubuntu-container-disk-v1.35.tag
  • packages/apps/kubernetes/images/ubuntu-container-disk/Dockerfile
  • packages/apps/kubernetes/templates/_helpers.tpl
  • packages/apps/kubernetes/templates/cluster.yaml
  • packages/apps/kubernetes/templates/helmreleases/ouroboros.yaml
  • packages/apps/kubernetes/templates/talos/bootstrap-token-tenant-job.yaml
  • packages/apps/kubernetes/templates/talos/talos-pki.yaml
  • packages/apps/kubernetes/tests/kubelet_reservation_test.yaml
  • packages/apps/kubernetes/values.schema.json
  • packages/apps/kubernetes/values.yaml
  • packages/system/capi-providers-cpprovider/Makefile
  • packages/system/capi-providers-cpprovider/files/control-plane-components.yaml
  • packages/system/capi-providers-cpprovider/images/cluster-api-control-plane-provider-kamaji/Dockerfile
  • packages/system/capi-providers-cpprovider/templates/configmaps.yaml
  • packages/system/capi-providers-cpprovider/templates/providers.yaml
  • packages/system/kubernetes-rd/cozyrds/kubernetes.yaml
💤 Files with no reviewable changes (8)
  • packages/apps/kubernetes/images/ubuntu-container-disk-v1.35.tag
  • packages/apps/kubernetes/images/ubuntu-container-disk-v1.31.tag
  • packages/apps/kubernetes/images/ubuntu-container-disk-v1.32.tag
  • packages/apps/kubernetes/images/ubuntu-container-disk-v1.33.tag
  • packages/apps/kubernetes/images/ubuntu-container-disk-v1.34.tag
  • packages/apps/kubernetes/images/ubuntu-container-disk-v1.30.tag
  • packages/apps/kubernetes/files/versions.yaml
  • packages/apps/kubernetes/images/ubuntu-container-disk/Dockerfile
✅ Files skipped from review due to trivial changes (3)
  • packages/apps/kubernetes/images/talos-csr-signer.tag
  • packages/apps/kubernetes/images/kubectl.tag
  • packages/apps/kubernetes/templates/_helpers.tpl
🚧 Files skipped from review as they are similar to previous changes (1)
  • packages/system/capi-providers-cpprovider/templates/configmaps.yaml

Comment thread packages/apps/kubernetes/images/talos-csr-signer/Dockerfile
Comment thread packages/apps/kubernetes/templates/cluster.yaml Outdated
Comment thread packages/apps/kubernetes/templates/talos/bootstrap-token-tenant-job.yaml Outdated

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

♻️ Duplicate comments (1)
packages/apps/kubernetes/templates/cluster.yaml (1)

644-644: ⚠️ Potential issue | 🟠 Major | ⚡ Quick win

dnsDomain is rendered inside the range $groupName, $group := .Values.nodeGroups loop, so . is the nodeGroup, not the chart root.

The kubernetes.tenantClusterDomain helper expects the chart root context (it reads .Values.…). With . being the nodeGroup map, the helper either fails or silently resolves to an empty/default cluster domain, which would mean Talos workers configure the kubelet with the wrong DNS domain and DNS lookups against *.svc.<tenantClusterDomain> would diverge from what Kamaji/CoreDNS expects. Use $ (root) here, consistent with line 293 where . happens to equal $ because it's outside the range.

🐛 Proposed fix
-            dnsDomain: {{ include "kubernetes.tenantClusterDomain" . }}
+            dnsDomain: {{ include "kubernetes.tenantClusterDomain" $ }}
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/apps/kubernetes/templates/cluster.yaml` at line 644, The dnsDomain
value is being rendered inside the range loop so "." is the nodeGroup rather
than the chart root, causing the helper kubernetes.tenantClusterDomain to read
the wrong context; update the template to call the helper with the root context
(use $) instead of . (i.e., replace the current dnsDomain: {{ include
"kubernetes.tenantClusterDomain" . }} with a call that passes $), ensuring the
helper reads .Values from the chart root when rendering dnsDomain for the
nodeGroup.
🧹 Nitpick comments (3)
packages/system/capi-providers-cpprovider/images/cluster-api-control-plane-provider-kamaji/Dockerfile (1)

22-23: ⚡ Quick win

Add --fail to curl and enable pipefail so download errors abort the build.

Without --fail, curl -sSL swallows non-2xx HTTP responses; combined with no set -o pipefail on the default /bin/sh -c, the RUN can silently succeed even when the upstream tag fetch fails or the body is empty. The downstream git apply and go build would then run on whatever happens to be in the working dir (or hard-fail with a confusing diagnostic). Belt-and-braces this so the failure surface is the actual curl error.

♻️ Proposed fix
-RUN curl -sSL "https://github.com/clastix/cluster-api-control-plane-provider-kamaji/archive/refs/tags/${VERSION}.tar.gz" \
-    | tar -xzf - --strip=1
+SHELL ["/bin/sh", "-eo", "pipefail", "-c"]
+RUN curl --fail -sSL "https://github.com/clastix/cluster-api-control-plane-provider-kamaji/archive/refs/tags/${VERSION}.tar.gz" \
+    | tar -xzf - --strip=1
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In
`@packages/system/capi-providers-cpprovider/images/cluster-api-control-plane-provider-kamaji/Dockerfile`
around lines 22 - 23, The RUN that streams the tarball via curl into tar should
be hardened: add curl's --fail flag and enable pipefail so that non-2xx
responses or broken pipes cause the Docker build to fail. Update the RUN that
uses curl (the line invoking curl -sSL "...${VERSION}..." | tar -xzf -
--strip=1) to run in a shell that sets -o pipefail (and -e) and to pass --fail
to curl; this ensures errors are surfaced before subsequent commands like git
apply or go build run on bad input.
packages/system/capi-providers-cpprovider/Makefile (2)

43-44: ⚡ Quick win

Use curl --fail so a broken UPSTREAM_VERSION doesn't pass through to tar.

Same pattern as the Dockerfile: curl -sSL succeeds on HTTP errors and pipes whatever body it got into tar. If a future bump renames a tag or the registry hiccups, you'll get a tar decompression error instead of a clear "release not found" message, and make update may go on to commit/apply patches against partial sources.

♻️ Proposed fix
-	curl -sSL "https://github.com/clastix/cluster-api-control-plane-provider-kamaji/archive/refs/tags/$(UPSTREAM_VERSION).tar.gz" \
-		| tar -xzf - --strip=1 -C .update-tmp
+	set -o pipefail; curl --fail -sSL "https://github.com/clastix/cluster-api-control-plane-provider-kamaji/archive/refs/tags/$(UPSTREAM_VERSION).tar.gz" \
+		| tar -xzf - --strip=1 -C .update-tmp
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/system/capi-providers-cpprovider/Makefile` around lines 43 - 44, The
curl invocation in the Makefile (the line invoking curl -sSL
"https://github.com/.../$(UPSTREAM_VERSION).tar.gz" | tar -xzf - --strip=1 -C
.update-tmp) should add the --fail flag so HTTP errors cause curl to exit
non‑zero instead of streaming an error page into tar; update that curl command
to include --fail (or -f) alongside -sSL to ensure make update fails immediately
when the release/tag is not found.

27-33: 💤 Low value

Fold the metadata-JSON cleanup into the shell chain so failures don't leave stale artifacts.

Recipe lines 27–32 run in one shell; line 33 is a separate sub-shell that only fires when the previous line succeeded. If the digest/sed/gzip chain fails (e.g., the containerimage.digest key isn't in the metadata), the JSON sticks around and the next make image may pick up a stale value. Moving the cleanup inside the chain (or trapping on EXIT) makes the target idempotent.

♻️ Proposed fix
 	digest=$$(yq e '."containerimage.digest"' images/cluster-api-control-plane-provider-kamaji.json -o json -r); \
+	trap 'rm -f images/cluster-api-control-plane-provider-kamaji.json' EXIT; \
 	IMG="$(IMAGE_BASE):$(IMAGE_TAG)@$$digest"; \
 	echo "$$IMG" > images/cluster-api-control-plane-provider-kamaji.tag; \
 	sed -i.bak -E "s|image: ghcr\.io/.+/cluster-api-control-plane-provider-kamaji:[^[:space:]]+|image: $$IMG|" files/control-plane-components.yaml; \
 	rm files/control-plane-components.yaml.bak; \
 	gzip -nc files/control-plane-components.yaml > files/components.gz
-	rm -f images/cluster-api-control-plane-provider-kamaji.json
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/system/capi-providers-cpprovider/Makefile` around lines 27 - 33, The
Makefile target leaves images/cluster-api-control-plane-provider-kamaji.json
behind if the digest/sed/gzip chain fails because the final rm is executed in a
separate shell; fold the cleanup into the same shell chain or add an EXIT trap
so removal always runs. Update the block that defines digest=$$(yq e ...),
IMG=... and the subsequent sed/gzip pipeline (the commands manipulating
images/cluster-api-control-plane-provider-kamaji.json and
files/control-plane-components.yaml) to either append the rm
images/cluster-api-control-plane-provider-kamaji.json into that same chained
line (so it runs even on failures using ; or || true as appropriate) or install
a trap 'rm -f images/cluster-api-control-plane-provider-kamaji.json' at the
start of the shell so the JSON is removed on exit; ensure you reference the
exact symbols: digest variable, IMG variable, sed -i.bak invocation, gzip -nc,
and rm images/cluster-api-control-plane-provider-kamaji.json so the cleanup is
always executed.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Duplicate comments:
In `@packages/apps/kubernetes/templates/cluster.yaml`:
- Line 644: The dnsDomain value is being rendered inside the range loop so "."
is the nodeGroup rather than the chart root, causing the helper
kubernetes.tenantClusterDomain to read the wrong context; update the template to
call the helper with the root context (use $) instead of . (i.e., replace the
current dnsDomain: {{ include "kubernetes.tenantClusterDomain" . }} with a call
that passes $), ensuring the helper reads .Values from the chart root when
rendering dnsDomain for the nodeGroup.

---

Nitpick comments:
In
`@packages/system/capi-providers-cpprovider/images/cluster-api-control-plane-provider-kamaji/Dockerfile`:
- Around line 22-23: The RUN that streams the tarball via curl into tar should
be hardened: add curl's --fail flag and enable pipefail so that non-2xx
responses or broken pipes cause the Docker build to fail. Update the RUN that
uses curl (the line invoking curl -sSL "...${VERSION}..." | tar -xzf -
--strip=1) to run in a shell that sets -o pipefail (and -e) and to pass --fail
to curl; this ensures errors are surfaced before subsequent commands like git
apply or go build run on bad input.

In `@packages/system/capi-providers-cpprovider/Makefile`:
- Around line 43-44: The curl invocation in the Makefile (the line invoking curl
-sSL "https://github.com/.../$(UPSTREAM_VERSION).tar.gz" | tar -xzf - --strip=1
-C .update-tmp) should add the --fail flag so HTTP errors cause curl to exit
non‑zero instead of streaming an error page into tar; update that curl command
to include --fail (or -f) alongside -sSL to ensure make update fails immediately
when the release/tag is not found.
- Around line 27-33: The Makefile target leaves
images/cluster-api-control-plane-provider-kamaji.json behind if the
digest/sed/gzip chain fails because the final rm is executed in a separate
shell; fold the cleanup into the same shell chain or add an EXIT trap so removal
always runs. Update the block that defines digest=$$(yq e ...), IMG=... and the
subsequent sed/gzip pipeline (the commands manipulating
images/cluster-api-control-plane-provider-kamaji.json and
files/control-plane-components.yaml) to either append the rm
images/cluster-api-control-plane-provider-kamaji.json into that same chained
line (so it runs even on failures using ; or || true as appropriate) or install
a trap 'rm -f images/cluster-api-control-plane-provider-kamaji.json' at the
start of the shell so the JSON is removed on exit; ensure you reference the
exact symbols: digest variable, IMG variable, sed -i.bak invocation, gzip -nc,
and rm images/cluster-api-control-plane-provider-kamaji.json so the cleanup is
always executed.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: fd70aea8-5fd5-47f9-b98c-452c9062c5b0

📥 Commits

Reviewing files that changed from the base of the PR and between 630c952 and e184ac6.

⛔ Files ignored due to path filters (1)
  • packages/system/capi-providers-cpprovider/files/components.gz is excluded by !**/*.gz
📒 Files selected for processing (7)
  • packages/apps/kubernetes/templates/cluster.yaml
  • packages/system/capi-providers-cpprovider/Makefile
  • packages/system/capi-providers-cpprovider/files/control-plane-components.yaml
  • packages/system/capi-providers-cpprovider/images/cluster-api-control-plane-provider-kamaji/Dockerfile
  • packages/system/capi-providers-cpprovider/images/cluster-api-control-plane-provider-kamaji/patches/0001-backport-additional-service-ports-pr347.diff
  • packages/system/capi-providers-cpprovider/templates/configmaps.yaml
  • packages/system/capi-providers-cpprovider/templates/providers.yaml

@myasnikovdaniil

Copy link
Copy Markdown
Contributor

Andrei Kvapil (@kvaps)

Per the official Talos support matrix, Talos v1.13 supports Kubernetes 1.31, 1.32, 1.33, 1.34, 1.35 and 1.36 — not just 1.32–1.34: https://docs.siderolabs.com/talos/v1.13/getting-started/support-matrix
my bad, you are right. Lets remove 1.30 from supported version first, and then ship new talos workers. I think we could add this to 1.4 before it released. it removes unnesessary support of older 1.12 talos or blocking users that used 1.30 k8s in time of upgrade to talos.

What about other changes that were requested?

@myasnikovdaniil myasnikovdaniil added the do-not-merge/hold Indicates that a PR should not merge because someone has issued /hold label May 15, 2026

@myasnikovdaniil myasnikovdaniil left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-review. Most of the prior feedback is addressed — Talos↔K8s support-matrix guard, v1.30 removed from Version enum/files/versions.yaml/kubernetes-rd CRD schema, kamaji control-plane provider rebuilt from upstream v0.19.0 (image digest-pinned, version labels aligned), talos-csr-signer sidecar hardened (resources + non-root/RO-rootfs/drop-all-caps/seccomp), talos.version/talos.schematicID exposed as values, pod/service CIDRs collapsed into top-of-template constants, four duplicated lookups hoisted above the nodeGroup loop, images.kubectl override + tightened bootstrap-token Job retry budget, wipe: false annotated, dead ubuntu-container-disk build chain removed, every bot thread now has a reply. Destructive-upgrade hazard from the last pass is gone.

Two non-blocking nits worth folding into a follow-up:

  • packages/apps/kubernetes/templates/talos/bootstrap-token-tenant-job.yaml:55-58TENANT_KUBECONFIG env var is set but never read (only KUBECONFIG matters to kubectl, and the inline shell doesn't reference it). May be drop it?

One acknowledged trade-off: tests/kubelet_reservation_test.yaml keeps the failure-path coverage but the happy-path TCT assertions from the kubeadm-era file weren't ported (the in-file comment explains why — TCT data is a YAML-encoded string, Secret mocking is left as a follow-up). Please open an issue so this doesn't drop off the radar.

LGTM.

The TalosConfigTemplate rendered by the kubernetes app chart had its
fields placed directly under spec instead of under spec.template.spec.
CABPT (bootstrap.cluster.x-k8s.io/v1alpha3) requires the standard
CAPI template wrapper and rejects the resource with:

    TalosConfigTemplate.bootstrap.cluster.x-k8s.io
    "<release>-<group>" is invalid: spec.template: Required value

This was introduced when the original Talos-bootstrap commit replaced
the KubeadmConfigTemplate block by removing the kubeadm spec.template
wrapper but did not re-add the equivalent for the new resource. Helm
template still rendered the document because YAML accepts the
fields-under-spec shape as valid YAML, but the API server admission
rejects it on apply. Manifested as: HelmRelease succeeds installing
every other resource, TalosConfigTemplate creation fails, CAPI never
creates Machines, no Talos worker boots.

Caught while exercising the full Phase 1 install path end-to-end on
a dev cluster: the chart-rendered manifest applies cleanly until CABPT
admission, which rolls the HR back. With the wrapper restored, TCT
admits successfully and the MachineDeployment.bootstrap.configRef
resolves to a valid template.

Existing helm unittest suite did not catch this because it only
asserts on TCT presence and on individual field values via
documentIndex, not on the resource's spec.template invariant. A
follow-up assertion `spec.template.spec.generateType == none` would
prevent regressions.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
The test script (tests/containerd-config-path-test.sh) plus its two
toml fixtures (tests/fixtures/containerd-{1x,2x}.toml) guarded a pair
of sed lines that lived inside the kubeadm-era KubeadmConfigTemplate's
preKubeadmCommands. That whole block disappeared when the chart
switched to TalosConfigTemplate — there are no preKubeadmCommands in
the Talos worker bootstrap path, containerd is configured by Talos
itself via its machineconfig dialect (machine.files / machine.kubelet
extraConfig), not by post-boot shell sed.

After this PR the script's grep -qF against the template fails with
`sed line drifted` because the strings it's looking for are simply
gone, taking down the chart's `make test` and the repo-level
`helm-unit-tests` CI step.

Remove the script, its Makefile invocation, and the now-orphan
fixtures directory. If a Talos-equivalent containerd assertion is
desired later it belongs in helm unittest against the rendered
machineconfig blob, not as a regex check on cluster.yaml.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
The image-talos-csr-signer target still used the legacy `$(call settag,
...)` helper, which was removed from hack/common-envs.mk in main and
replaced by the `$(call image-tags, <repo>, <versioned-tag>)` macro
used by every other image-* target in this Makefile. After rebasing
onto current main the unresolved `settag` call expands to an empty
string, so the build emits `--tag <REGISTRY>/talos-csr-signer:` with
nothing after the colon and docker buildx rejects it with
`invalid reference format`. CI Build job exits 1.

Re-aligning with image-cluster-autoscaler / image-kubevirt-csi-driver
which already use image-tags. The follow-up `echo` line that pins the
digest into images/talos-csr-signer.tag now references $(IMAGE_TAG)
directly, matching the same convention.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
…idecar boot

The e2e fixture's `kubectl wait` budgets were tuned to the pre-Talos
Kamaji cold-start (3-4 minutes for kube-apiserver+controller-manager+
scheduler+konnectivity). After this PR the apiserver pod additionally
pulls and starts the talos-csr-signer sidecar, and cert-manager must
issue three Talos PKI Certificates (talos-ca, talos-tls-cert, plus the
Kamaji-issued ca) before the wait-for-kubeconfig init container clears.
Cold-start in a fresh CI sandbox (no warm image cache) consistently
crosses the old 4m/5m budgets and the test fails at
`kubectl wait tcp ... --timeout=5m` with
`error: timed out waiting for the condition on tenantcontrolplanes/<name>`.

Raise both `TenantControlPlaneCreated` and `tcp ... version.status=Ready`
budgets to 10m, matching the `release.cozystack.io/helm-install-timeout: 15m`
that cozystack-api stamps onto every Kubernetes tenant HR. Downstream
waits (`deploy --timeout=4m`, `machinedeployment --timeout=1m`) are
unaffected by Talos PKI work and keep their tighter budgets.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
…admission accepts the unmodified install

The MachineHealthCheck CRD's `spec.maxUnhealthy` is an IntOrString. The
CAPI admission webhook accepts either a bare integer ("0", "1", ...) or
a percentage ("0%", "50%") but rejects a bare numeric string with:

    admission webhook "validation.machinehealthcheck.cluster.x-k8s.io"
    denied the request: MachineHealthCheck ... is invalid:
    spec.maxUnhealthy: Invalid value: intstr.IntOrString{...,
    StrVal:"0"}: must be either an int or a percentage:
    invalid value for IntOrString: invalid type: string is not a
    percentage

The previous default ("0", a string without a percent sign) hit exactly
that path and caused Helm install to fail on every fresh tenant
Kubernetes install — surfaced by the e2e suite's
`Create a tenant Kubernetes control plane with latest version` step,
which then dropped the chart (cascade-deleted KamajiControlPlane) and
the downstream wait never had anything to find.

"0%" is semantically identical (any unhealthy node triggers remediation
immediately) and passes admission. Custom values like "50%" or bare
integers like "1" continue to work; the field stays parametric for
operators that want to relax remediation during a kubeadm-to-Talos
rollover.

Regenerated values.schema.json, README.md, types.go, and the
kubernetes-rd cozyrds OpenAPI schema via the chart's `make generate`.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
PR review on this branch flagged that the previous default ("0" string)
was still wrong even after the values.yaml default flip: any operator
that overrides nodeHealthCheck.maxUnhealthy to a bare integer (e.g. "1",
which is also a valid IntOrString) gets the same admission rejection,
because `| quote` unconditionally wraps the rendered value in quotes
and the MHC webhook treats "1" as a non-percentage string.

Render conditionally: if the supplied value has a `%` suffix, keep the
existing quoted-string form (percentages must be strings). Otherwise,
render as a bare integer via the Sprig `int` function — that handles
the chart-default "0%", the now-safe "0", and any operator override
like "1" or "5" all in one path.

Follow-up to PR review:
cozystack#2610 (comment)

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
… Helm template into post-install hook Job

Chart used Helm `lookup` to gate the TalosConfigTemplate (TCT) and to
fold the apiserver Service ClusterIP into KamajiControlPlane's
network.certSANs. Both inputs — the Service, the Talos-CA TLS Secret,
the Kamaji-issued tenant CA Secret — only exist AFTER Helm applies the
first render: Kamaji controller emits the Service in reaction to the
KamajiControlPlane CR this same chart applies, cert-manager issues the
Talos PKI from the Certificate CR this same chart applies, and Kamaji
generates the tenant CA as part of its bootstrap. On a fresh tenant
install all three lookups return nil on first render, $talosReady is
false, TCT is skipped, and the certSANs list omits the ClusterIP.

Helm 3's release-storage three-way merge then refuses to upgrade
because no values changed. The lookups become valid 30s later, but
nothing requests a re-render. The MachineDeployment.bootstrap.configRef
points at a TalosConfigTemplate that never lands, so CAPI's
MachineDeployment controller emits `cannot create a new MachineSet
when templates do not exist` forever. The apiserver TLS cert is
generated by Kamaji without the live ClusterIP in its SAN list, so
even if a worker were to boot, its kubelet would reject the apiserver
on `x509: certificate is valid for <DNS> not <IP>`.

Suspend+resume and values-drift tricks force a second render manually
but every fresh install hits the same race; that's not a chart-level
fix, it's a workaround the operator has to remember on every tenant.

Move both objects into a post-install/post-upgrade Helm hook Job
(templates/talos/talos-reconcile-job.yaml) that:
  - waits up to ~10m for Service.spec.clusterIP, <release>-talos-ca,
    <release>-ca to be observable;
  - JSON-merge-patches KamajiControlPlane.spec.network.certSANs to
    include the live ClusterIP plus the two Service DNS names (Kamaji
    re-issues the apiserver TLS cert in response);
  - reads stable random tokens from <release>-talos-secrets (chart-self
    rendered, always present at hook time);
  - kubectl applies the TalosConfigTemplate with the full machineconfig
    inlined and an ownerReference to the KamajiControlPlane.

The owner reference lets the Kubernetes garbage collector reap the TCT
when the chart's KCP is deleted on `helm uninstall`; no pre-delete
hook needed. The Job runs at hook-weight 5, between the chart's
post-install resources and the bootstrap-token-tenant-job at weight 10,
so the bootstrap-token Secret lands inside the tenant kube-system only
after the worker machineconfig that consumes it is in place.

cluster.yaml drops the lookup gates ($apiSvc / $apiSvcIP /
$talosSecrets / $talosCA / $k8sCA / $talosReady / $talosToken /
$clusterId / $clusterSecret / $bootstrapToken / $talosCACrt /
$k8sCACrt / $apiEndpoint) and the gated TCT block. The certSANs list
on the KamajiControlPlane keeps only the two DNS entries; the
ClusterIP arrives via the hook.

The e2e fixture (hack/e2e-apps/run-kubernetes.sh) waits 5m instead of
1m for `machinedeployment ... status.replicas=2` so the
post-install-hook + Talos image pull + first VM boot fit in budget on
a cold sandbox.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
… reservations

The post-install hook Job is the new home of the TalosConfigTemplate
authorship (replacing the lookup-and-reuse pattern). On any cozystack
tenant the default-deny CiliumClusterwideNetworkPolicy on the release
namespace blocks the hook pod from reaching the management apiserver
to read the Service ClusterIP / Talos CA / k8s CA / KamajiControlPlane
it has to reconcile, leaving the Job stuck in a wait loop and the
release pending-install.

Ship a CiliumNetworkPolicy alongside the hook that allows egress to
the kube-apiserver entity (and kube-dns for name resolution) for pods
labelled cozystack.io/talos-reconcile=<release>. Cilium policies are
additive, so the tenant default-deny stays in force for everything
else in the namespace.

Inline the kubelet-reservation computation from cluster.yaml directly
in the hook template; the previous helper reference was unresolved at
render time and aborted the chart. Numbers match cluster.yaml's
auto-computed system/kube reserved memory (5% of effective memory
clamped to 256Mi..1Gi) and CPU (5% clamped to 50m..500m).

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
…hook

The Talos PKI Certificate that secures the trustd / apid sidecar
endpoint at https://<Service ClusterIP>:50001 used the same
lookup-and-reuse pattern as the TalosConfigTemplate the previous fix
already moved out into the post-install hook. On a fresh install the
Service ClusterIP does not yet exist when the chart renders, the
lookup returns nil, and the Certificate is issued with only the
DNS-name SANs. Workers then fail TLS to apid with "cannot validate
certificate for <ClusterIP> because it doesn't contain any IP SANs"
and stall at "service apid to be up".

Drop the lookup gate from talos-pki.yaml and add a second
kubectl patch step to the talos-reconcile hook Job that writes
spec.ipAddresses on <release>-talos-tls-cert once the Service is
live. cert-manager reissues the cert with the new SAN, the worker's
TLS handshake succeeds, and apid comes up.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
…los worker migration

Seven distinct issues found during cozy-review and end-to-end testing
on a dev cluster. Bundled here as a single squashed commit:

* nodeHealthCheck.maxUnhealthy default 0% -> 50%.
  Talos workers boot in pulls from factory.talos.dev that exceed the
  10m nodeStartupTimeout on slow networks. With the prior 0% default a
  single Machine briefly NotReady triggers MHC remediation, CAPI spawns
  a replacement that also boots slowly, and the cluster loops. 50%
  leaves room for transient unhealthy nodes during the kubeadm-to-Talos
  rollover; drop back to 0% in a follow-up once fleets stabilise.

* bootstrap-token Job backoffLimit 5 -> 10.
  Aligns the Job's worst-case wall time (~15m under exponential backoff
  capped at 6m per retry) with the chart's HR install timeout
  (release.cozystack.io/helm-install-timeout: 15m). At backoffLimit=5
  the Job could exhaust retries before the tenant apiserver came up on
  a slow cluster, failing the chart upgrade silently.

* migration 44 (v1.30 -> v1.31) surfaces per-tenant failures.
  Tracks failed (ns, name) pairs and writes them to a
  cozystack.io/migration-44-failed-tenants annotation on the
  cozystack-version ConfigMap. Operators previously had to grep Job
  logs because the migration exits 0 (intentional, for retry) and the
  Job status stays green while individual tenants are stuck at v1.30.

* talos-reconcile-job memory limit 64Mi -> 256Mi.
  Reproduced OOMKill on dev3: pod completed the wait loop and the
  KamajiControlPlane / Certificate JSON-merge patches, then died at
  the kubectl-apply that pipes a TalosConfigTemplate manifest. Fresh
  installs survive the 64Mi limit because kubectl's working set is
  smaller against an empty cluster; upgrades on established tenants
  pay the merge-and-diff cost. 256Mi leaves headroom for the embedded
  JSON marshalling and discovery cache.

* Talos workers reach the apiserver via DNS instead of the live
  ClusterIP. The post-install hook previously patched the Talos PKI
  Certificate to add the apiserver Service ClusterIP as an IP SAN.
  cert-manager reissues the Secret but the csr-signer sidecar loads
  its TLS cert once at startup and does not watch the file, so on a
  fresh install it keeps serving the original (no-IP-SAN) cert.
  Talos apid then fails the trustd TLS handshake with 'x509: cannot
  validate certificate for <IP> because it does not contain any IP
  SANs' and the worker boot stalls on 'service apid to be up'
  indefinitely. Drop the certificate patch and route Talos workers
  to the apiserver via <release>.<namespace>.svc. The Talos
  machineconfig now sets cluster.controlPlane.endpoint to the DNS
  form and seeds machine.network.extraHostEntries with the live
  ClusterIP, so worker /etc/hosts resolves the name without depending
  on tenant CoreDNS (which is itself bootstrapped after Cilium, which
  dials this same endpoint). The Talos PKI Certificate already carries
  DNS SANs from chart render time so TLS verification succeeds against
  the chart-issued cert with no race. Cilium's k8sServiceHost can also
  drop the lookup-and-reuse fallback and use plain DNS.

* CABPT controller-manager runs under a uniquely-named ServiceAccount.
  The vendored CABPT components manifest uses the implicit 'default'
  SA in cabpt-system. cluster-api operator copies the manifest and
  rewrites every {kind: ServiceAccount, name: default, namespace:
  cabpt-system} subject to point at the BootstrapProvider's own
  namespace (cozy-cluster-api). The Deployment still runs as
  cabpt-system/default, the controller has no RBAC and CrashLoops on
  'cannot list resource ... at the cluster scope'. TalosConfigs never
  reconcile and Machines stay forever in WaitingForBootstrapData. The
  kubeadm bootstrap provider in the same chart works because it uses
  a uniquely-named SA the operator leaves alone. Mirror that pattern
  for CABPT: ServiceAccount cabpt-manager in cabpt-system, wire all
  four (Cluster)RoleBindings to it, and set the Deployment's
  serviceAccountName accordingly. Regenerated components-talos.gz
  from the edited source.

* Drop the GPU node-label test obsoleted by the Talos worker switch.
  Main added a test asserting GPU node-groups register their nodes
  with gpu=on via KubeadmConfigTemplate.spec.template.spec.joinConfiguration.nodeRegistration.kubeletExtraArgs.node-labels.
  After this PR, KubeadmConfigTemplate is replaced by
  TalosConfigTemplate rendered out-of-band by the talos-reconcile
  hook and the chart no longer carries the kubelet args the test
  inspects. The GPU labeling feature itself is not yet wired up on
  Talos workers in this Phase 1 (group.gpus still attaches GPU
  devices to the VM but nothing sets gpu=on on the resulting node).
  Tracking as a Phase 1 regression rather than restoring the
  now-irrelevant kubeadm-shaped assertion.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
…olver

Tenant Kubernetes workers in main run as Ubuntu+kubeadm KubeVirt VMs
and inherit pod DNS via cloud-init, so they resolve management-cluster
services (NFS endpoints used by kubevirt-csi like
linstor-csi-nfs.cozy-linstor.svc) out of the box. Talos workers in this
PR are strict and consult only what is in machineconfig; with no host
resolver wired up, kubelet's NFS mount syscall fails with NXDOMAIN
when kubevirt-csi-backed PVCs are mounted.

The reconcile hook now looks up the management CoreDNS Service ClusterIP
at runtime and injects it into machine.network.nameservers in the
TalosConfigTemplate. Tenant in-cluster pods are unaffected: they use
kubelet --cluster-dns (tenant CoreDNS) via dnsPolicy: ClusterFirst.
This nameserver only covers host-side paths (mounts, image pulls,
pods that opt out of cluster DNS), which is what main provided via
cloud-init.

Adds a narrow ClusterRole + ClusterRoleBinding scoped to
resourceNames: ["kube-dns"] so the hook ServiceAccount can read the
single well-known Service without broader cluster-scoped privilege.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
Follows cozystack#2855. The talos-csr-signer image target was added on this
branch before cozystack#2855 merged, so the rebase brought it in still
pointing --cache-from at $(REGISTRY)/talos-csr-signer:latest, which
on PR CI is the per-CI registry where :latest is never published.
That cache lookup 404s on every PR build and forces a cold rebuild,
exactly the failure mode cozystack#2855 fixed for the other Makefiles.

Aligns the only remaining $(REGISTRY)-targeted --cache-from with the
$(CACHE_REGISTRY) ghcr.io convention from cozystack#2855 so PR builds of this
target warm-start off the last release's :latest.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
@IvanHunters
IvanHunters force-pushed the phase1-talos-migration branch from db35e94 to 27dbca7 Compare June 9, 2026 09:53
Follow-up to the CoreDNS nameserver injection. Pointing the worker's
host resolver at management CoreDNS is necessary but not sufficient:
kubevirt-csi calls `mount.nfs linstor-csi-nfs.cozy-linstor.svc` with
no domain suffix, and management CoreDNS only knows the full FQDN
(<svc>.<ns>.svc.<management-cluster-domain>). Without search domains
in /etc/resolv.conf the resolver hands the partial name to CoreDNS
verbatim and gets NXDOMAIN — observed in CI as the NFS mount step
failing roughly 20 minutes into the E2E run.

Ubuntu+kubeadm workers in main get these search domains for free
from cloud-init/DHCP in the management pod network. Talos reads
only what is in machineconfig, so this commit mirrors that.

The chart sources management cluster domain from
.Values._cluster["cluster-domain"] (populated by cozystack-controller
from the platform config; default cozy.local). cluster.local is
appended as a fallback when the management domain differs, covering
sandbox/CI deployments that still ship with the Kubernetes default.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

NOT LGTM — the architecture is sound and well-documented inline, but there are functional regressions (GPU node groups, air-gapped registry mirrors), two defects in the platform migrations, and a shipped-artifact mismatch that need fixing before merge.

Business context: Phase 1 of the Kubernetes-app split (cozystack/community#8) — replace the Ubuntu+kubeadm tenant worker bootstrap with Talos via CABPT and a talos-csr-signer sidecar, rolling existing tenants over automatically.

Blockers

B1: Migrations 44/45 strip the deletion-protection label from cozystack-version

File: packages/core/platform/images/migrations/migrations/44:104-106, 45:110-112

Issue: Both migrations stamp the version via bare kubectl create configmap ... --dry-run=client -o yaml | kubectl apply -f -, which emits no labels.

Evidence: Migration 43 carries the labeled manifest inline with an explicit comment that a label-less apply by the same field manager strips the platform.cozystack.io/no-delete label; templates/cozystack-version.yaml only re-renders the labeled ConfigMap on first install. The client-side three-way merge removes the label, dropping the version anchor out of the cozystack-no-delete-guardrail ValidatingAdmissionPolicy.

Impact: After the first upgrade through these migrations, cozystack-version is silently unprotected against deletion. E2E does not exercise this path (fresh installs skip migrations).

Fix: Mirror migration 43's labeled heredoc manifest in both stamps.

B2: Migration 44 stamps the wrong version and its retry design is defeated by migration 45

File: packages/core/platform/images/migrations/migrations/44:105

Issue: The file is named 44 (runs when current version is 44) but stamps version=44 instead of 45; its failure-path comment says "leaving CURRENT_VERSION at 43". Worse: on partial failure migration 44 exits 0 without stamping, but run-migrations.sh then continues to migration 45, which on success stamps version=46 — so migration 44 never retries, defeating its own best-effort design.

Evidence: run-migrations.sh:34-46 iterates seq CURRENT TARGET-1 unconditionally on exit 0; migration 45 unconditionally stamps 46 on its own success.

Impact: If any kubectl annotate in 44 fails, the keep-annotation pass is silently skipped forever, and Helm may prune KubeadmConfigTemplates still referenced by pre-rollover MachineSets — the exact hazard migration 44 exists to prevent.

Fix: Stamp 45 in migration 44, fix the comments, and make a partial failure actually stop the chain (or otherwise guarantee a retry) so the best-effort semantics hold.

B3: GPU node groups lose the gpu=on node label — GPUs stop being exposed on Talos workers

File: packages/apps/kubernetes/templates/talos/talos-reconcile-job.yaml (worker machineconfig)

Issue: The old KubeadmConfigTemplate set node-labels: "gpu=on" for groups with gpus; the Talos machineconfig has no machine.nodeLabels at all.

Evidence: main's cluster.yaml:482-489 (label + rationale comment); the HAMi device-plugin DaemonSet selects nodes via nvidiaNodeSelector: gpu: "on" (packages/system/hami/charts/hami/values.yaml:352-353, not overridden by the in-tenant HelmRelease). The pinning test tests/gpu_node_labels_test.yaml was deleted in this PR rather than ported.

Impact: On Talos workers the HAMi device plugin stays at DESIRED=0 and no nvidia.com/* resources are advertised — a functional regression for every GPU tenant.

Fix: Add machine.nodeLabels: {gpu: "on"} for groups with gpus and restore the test.

B4: Air-gapped/registry-mirror support for tenant workers silently regresses

Files: packages/apps/kubernetes/templates/cluster.yaml:101-102 (DataVolume URL), talos-reconcile-job.yaml (install.image), templates/copy-patch-containerd.yaml (now orphaned)

Issue: Three related drops: (1) the worker boot image is fetched by CDI over HTTPS from a hardcoded https://factory.talos.dev/... with no override value; (2) install.image: factory.talos.dev/installer/... is likewise hardcoded and the machineconfig has no machine.registries block, so it cannot be mirrored; (3) the patch-containerd Secret is still copied into the tenant namespace but its only consumer (the containerd certs.d mount in the kubeadm template) was deleted — the platform registries.mirrors feature no longer reaches tenant workers.

Evidence: Pre-PR workers booted from an OCI containerDisk (main cluster.yaml:115-116), mirrorable via containerd mirrors; the public air-gapped install guide documents tenant-worker registry mirrors via patch-containerd as a supported feature; nothing consumes the copied Secret anymore and the new machineconfig has no machine.registries. The nodeHealthCheck field docs even advertise "air-gapped mirrors of factory.talos.dev", which nothing in the chart can configure.

Impact: Tenant clusters in air-gapped environments cannot create Talos workers at all (image fetch fails), and operators using registries.mirrors lose mirrors on tenant workers without any error.

Fix: Expose the image URL (and installer ref) as values defaulting to factory.talos.dev, and plumb patch-containerd/platform registries into machine.registries.mirrors — or explicitly document the loss and remove the dead copy template.

B5: diskSize and related docs now describe removed mechanics

Files: packages/apps/kubernetes/values.yaml:84, api/apps/v1alpha1/kubernetes/types.go (DiskSize), README.md (migration-41 paragraph)

Issue: "Persistent disk size for kubelet and containerd data" — on Talos this is the system disk imaged from the factory artifact; the separate kubelet disk and its persistence semantics were removed in this PR. The README paragraph about state persisting across same-VM reboots describes the deleted layout.

Evidence: the branch drops the kubelet disk and cluster.yaml now provisions a single disk-system DataVolume.

Fix: Update field descriptions and README, regenerate the schema/RD.

B6: New code paths land without tests

Files: pkg/config/config.go (ParseHelmInstallDisableWaitAnnotation), packages/apps/kubernetes/templates/talos/*

Issue: (a) The new annotation parser has zero tests while its sibling ParseHelmInstallTimeoutAnnotation has a table test in pkg/config/config_test.go:18; the HelmInstallDisableWait plumbing in rest.go is likewise untested. (b) None of the four new talos templates has helm-unittest coverage — including two behaviors that were actual bugs fixed during this branch (MHC maxUnhealthy int-vs-percent rendering, the Talos-to-Kubernetes support-matrix fail guard) and the kubelet-reservation math duplicated between cluster.yaml and the reconcile hook, which will drift unnoticed.

Fix: Add the parser table test, MHC rendering tests, support-matrix guard tests, and minimal assertions on the talos templates.

B7: talos-csr-signer.tag contradicts the build-from-source chain

File: packages/apps/kubernetes/images/talos-csr-signer.tag

Issue: The tag pins ghcr.io/clastix/talos-csr-signer:latest@sha256:827b... (upstream), while the Dockerfile/Makefile added in this PR build the binary from a pinned commit into $(REGISTRY)/talos-csr-signer:$(IMAGE_TAG)@<digest> — the regenerated tag was never committed, so the chart ships the upstream image and the from-source supply chain is dead weight.

Evidence: packages/apps/kubernetes/Makefile:22-31 writes $(REGISTRY)/talos-csr-signer:... into the tag file; compare the cpprovider counterpart, which did get its regenerated tag (ghcr.io/cozystack/cozystack/cluster-api-control-plane-provider-kamaji:v0.19.0-cozystack.0@sha256:...).

Fix: Run the build chain and commit the regenerated tag; also fix the Dockerfile comment claiming :<commit> tagging (the Makefile tags by chart version).

B8: Stale comments after the 15m-to-20m timeout bump and the TCT move

Files: templates/_helpers.tpl:77, templates/talos/bootstrap-token-tenant-job.yaml:43-44, hack/e2e-apps/run-kubernetes.sh:111, tests/cluster_test.yaml header, tests/kubelet_reservation_test.yaml header, cluster.yaml:33-39

Issue: Multiple comments still reference the 15m install timeout (now 20m via the cozyrds annotation), the removed $talosReady lookup gate, and the removed hoisted-lookups block.

Fix: Sweep and update; these comments are load-bearing for the next person debugging hook timing.

B9: Dead code left from abandoned approaches

Files: cluster.yaml:52-54 (unused $kubeletVersion/$talosVersion/$talosSchematic), talos-reconcile-job.yaml:82-85 (leftover cert-manager.io/certificates get,patch RBAC from the dropped IP-SAN patching approach — an unnecessary grant), bootstrap-token-tenant-job.yaml:58-59 (unused TENANT_KUBECONFIG env)

Fix: Remove all three; the RBAC one also narrows the hook's privileges to what it actually uses.

Non-blocking follow-ups

  1. The kubeadm-format bootstrap token is created without expiration (bootstrap-token-tenant-job.yaml:87-101) — a permanent credential where CABPK previously rotated short-lived tokens; TALOS_TOKEN/CLUSTER_SECRET/bootstrap token also sit in plaintext in the TalosConfigTemplate (a non-Secret CR). The reconcile hook already re-runs on every upgrade and is a natural rotation point.
  2. Deduplicate the ~45 lines of kubelet-reservation math mirrored between cluster.yaml and talos-reconcile-job.yaml into a shared named template.
  3. TalosConfigTemplates of removed node groups accumulate until KCP deletion (owner-ref GC only cascades from the KamajiControlPlane); consider per-group cleanup in the hook.
  4. capi-providers-bootstrap has no update target documenting how bootstrap-components-talos.yaml/components-talos.gz were produced (the cabpt-manager SA edit is hand-applied to a vendored bundle); the cpprovider package has one — make it symmetric.
  5. Install/Upgrade.DisableWait changes what "Ready" means for the Kubernetes app HelmRelease (install success no longer implies addon readiness). The cozyrds comment explains why; worth a line in the release note so operators don't read HR Ready as "workers and addons up".

fi

kubectl create configmap --namespace cozy-system cozystack-version \
--from-literal version=44 --dry-run=client --output yaml \

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two problems with this stamp (B1/B2 in the review body): (1) a bare kubectl create | kubectl apply emits no labels, so the client-side three-way merge strips platform.cozystack.io/no-delete that migration 42 set and migration 43 deliberately carries inline — the ConfigMap falls out of the no-delete guardrail VAP. (2) This file runs when the current version is 44 (migration 43 already stamps 44), so it must stamp 45, not 44; the "leaving CURRENT_VERSION at 43" comment above is off by one too. Also note the partial-failure path (exit 0 without stamping) doesn't actually retry: the runner continues to migration 45, which stamps 46 on its own success, so a failed annotate pass here is silently skipped forever.

"cozystack.io/migration-45-failed-tenants-" >/dev/null 2>&1 || true

kubectl create configmap --namespace cozy-system cozystack-version \
--from-literal version=46 --dry-run=client --output yaml \

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same label issue as migration 44: this bare kubectl create | kubectl apply strips the platform.cozystack.io/no-delete label from cozystack-version. Please mirror migration 43's labeled heredoc manifest (it carries an explicit comment about exactly this trap).

- ${RELEASE}.${NS}.svc.{{ $dnsDomain }}
ca:
crt: ${TALOS_CA_B64}
kubelet:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The machineconfig is missing machine.nodeLabels for GPU node groups. The old KubeadmConfigTemplate set node-labels: "gpu=on" when the group has gpus, and the HAMi device-plugin DaemonSet schedules via nvidiaNodeSelector: gpu: "on" — without the label it stays at DESIRED=0 and no GPUs are advertised on Talos workers. The pinning test (tests/gpu_node_labels_test.yaml) was deleted rather than ported; please add machine.nodeLabels: {gpu: "on"} for GPU groups and restore the test.

ignores configdrive and never applies the machineconfig. */}}
source:
http:
url: https://factory.talos.dev/image/{{ $.Values.talos.schematicID }}/{{ $.Values.talos.version }}/openstack-amd64.raw.xz

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hardcoding https://factory.talos.dev/... (here and in install.image in the reconcile hook) with no override value breaks air-gapped environments: pre-PR the worker image was an OCI containerDisk and could be mirrored via containerd registry mirrors; a CDI HTTP source cannot. Combined with the now-unconsumed patch-containerd Secret (no machine.registries in the machineconfig), the documented tenant-worker registry-mirror feature silently stops working. Please expose the image URL/installer ref as values and plumb registry mirrors into machine.registries.mirrors — or document the loss explicitly and drop the dead copy template.

Comment thread pkg/config/config.go
// callers can leave HelmInstallDisableWait zero and let flux defaults
// apply.
func ParseHelmInstallDisableWaitAnnotation(raw string) (bool, error) {
switch raw {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ParseHelmInstallDisableWaitAnnotation has no tests while its sibling ParseHelmInstallTimeoutAnnotation has a table test in pkg/config/config_test.go. Please add the same table coverage (empty / true variants / false variants / garbage), plus a test for the HelmInstallDisableWait plumbing in convertApplicationToHelmRelease.

@@ -0,0 +1 @@
ghcr.io/clastix/talos-csr-signer:latest@sha256:827b62b5fc2859d66f06f5c1f8d2473ab7109d0600d551269d8ddb98e4a39a18

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This still pins the upstream ghcr.io/clastix/talos-csr-signer:latest@sha256:... image, while the Dockerfile/Makefile in this PR build from source into $(REGISTRY)/talos-csr-signer:$(IMAGE_TAG)@<digest> — so the chart actually ships the upstream binary and the from-source chain is dead weight. Compare the cpprovider tag, which did get regenerated (...cluster-api-control-plane-provider-kamaji:v0.19.0-cozystack.0@sha256:...). Please run the build chain and commit the regenerated tag; the Dockerfile comment about :<commit> tagging also doesn't match the Makefile (it tags by chart version).

# backoffLimit + restartPolicy: OnFailure gives the Job up to 11 attempts at
# the tenant apiserver with exponential backoff (capped at 6 min between
# restarts). Worst case ~15 min, aligned with the chart's HR install timeout
# (release.cozystack.io/helm-install-timeout: 15m on the cozyrds entry) so

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale comment: the install timeout was bumped to 20m in the cozyrds annotation, but this still says 15m (same in _helpers.tpl:77 and hack/e2e-apps/run-kubernetes.sh:111). Separately, non-blocking: the bootstrap-token Secret below is created without expiration, i.e. a permanent credential where CABPK previously rotated short-lived tokens — this hook re-runs on every upgrade and would be a natural rotation point.

…nd correct version

Two defects in the migration pair flagged in PR review:

(1) Both stamps used `kubectl create cm ... | kubectl apply -f -`, which
emits a label-less manifest. The same field manager (kubectl-client-side-
apply) then strips the platform.cozystack.io/no-delete label migration 42
carried on the cozystack-version ConfigMap, dropping it out of the
cozystack-no-delete-guardrail ValidatingAdmissionPolicy. After the first
upgrade through 44 or 45 the version anchor was silently unprotected
against deletion. The chart's templates/cozystack-version.yaml only
re-renders the labeled ConfigMap on first install, so the label has to
travel inline through every stamping migration. Mirrors the labeled
heredoc pattern migration 43 already uses for the same reason.

(2) Migration 44 stamped version=44 instead of 45 (it runs when current
version is 44 and advances to 45 — same convention as every other
migration in this directory). The mismatch was masked because migration
44's partial-failure path also did `exit 0` without stamping, while
migration 45 on success unconditionally stamped 46 — so a partial
failure in 44 silently left the keep-annotation pass undone, and the
"retry on next upgrade" comment in 44's header was untrue: run-
migrations.sh continued past it to migration 45, which advanced the
stamp past 44's retry window forever.

Best-effort semantics are not safe for migration 44 specifically: any
KubeadmConfigTemplate left un-pinned is at risk of being pruned by the
kubernetes chart upgrade, which then breaks the still-extant kubeadm-
backed MachineSet's bootstrap.configRef while the Talos rollover is
in flight. Switched to exit 1 on partial failure so run-migrations.sh
halts and the operator must investigate before the chart upgrade
proceeds. Migration 45 keeps its best-effort + FAILED_TENANTS
annotation pattern unchanged — bumping spec.version v1.30->v1.31 on
one tenant failing does not cascade into other tenants' chart upgrades.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
…ittest

The old KubeadmConfigTemplate set kubeletExtraArgs.node-labels="gpu=on"
for nodeGroups carrying .gpus; the Talos machineconfig the post-install
hook applies had no machine.nodeLabels at all. HAMi's hami-device-plugin
DaemonSet selects nodes via nvidiaNodeSelector: gpu: "on", so on Talos
workers the plugin stayed at DESIRED=0 and no nvidia.com/* resources
were advertised — a functional regression for every GPU tenant flagged
in PR review.

Adds machine.nodeLabels: {gpu: "on"} into the TalosConfigTemplate
emitted by templates/talos/talos-reconcile-job.yaml, gated on
$group.gpus to match main's per-group conditional. The inline comment
points the next reader at the HAMi selector so the contract is obvious.

Restores tests/gpu_node_labels_test.yaml — adapted from the deleted
KubeadmConfigTemplate-targeted version to assert on the rendered post-
install Job script via matchRegex (the TCT itself is not rendered by
helm; the hook applies it at runtime), covering both the "has gpus -->
labeled" and "no gpus --> no nodeLabels" paths.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
…apped regression

PR review flagged that `templates/copy-patch-containerd.yaml` still
copies the cozy-system/patch-containerd Secret into the tenant namespace
but its only consumer — the containerd certs.d mount in the deleted
kubeadm template — went away with the Talos rollover. The Talos
machineconfig the post-install hook applies has no machine.registries
block, no installer override, and the worker boot image is fetched
by CDI over HTTPS directly from factory.talos.dev. The chart-shipped
copy of the patch-containerd Secret is therefore unreferenced cruft
and the platform-wide registries.mirrors feature no longer reaches
tenant workers at all.

Phase 1 of the Talos migration is scoped to the rollover itself;
restoring air-gapped/mirror support for tenant workers via
machine.registries.mirrors (and matching imageBase/installerImage
override knobs) is non-trivial — containerd certs.d format does not
translate 1:1 to the Talos registries schema — and is deferred to a
follow-up. Drops the orphan template, removes the misleading
"air-gapped mirrors of factory.talos.dev" hint from the
nodeStartupTimeout doc (operators reading that today would assume
mirrors work — they do not), and adds an explicit Breaking Changes
entry to the chart README so air-gapped operators see this regression
before they upgrade rather than after a failed worker boot.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
PR review flagged three spots where dead code accumulated as the
branch iterated on the post-install hook design:

(1) cluster.yaml: $kubeletVersion / $talosVersion / $talosSchematic
were defined at the top of the per-loop template after the TCT was
moved into talos-reconcile-job.yaml — the hook redefines them in its
own context and the cluster.yaml block referenced none of them.

(2) talos-reconcile-job.yaml: the cert-manager.io/certificates get +
patch RBAC was added when the hook patched the Talos PKI Certificate's
spec.ipAddresses to include the apiserver Service ClusterIP. That
approach was dropped in favour of routing workers through the apiserver
Service DNS name (with extraHostEntries injecting the ClusterIP into
the worker /etc/hosts), which sidesteps the TLS race against the csr-
signer sidecar entirely. The RBAC grant is now unused and the hook is
now strictly Talos-scoped — narrowing it removes an unnecessary
cluster-write-adjacent grant for the hook ServiceAccount.

(3) bootstrap-token-tenant-job.yaml: TENANT_KUBECONFIG env var was a
holdover from an earlier design that distinguished the kubectl
kubeconfig from a tenant-only handle; the current script reads only
$KUBECONFIG (which already points at the same path), so the second
variable was unreferenced.

No behavioural change — the chart renders identically and helm-unittest
suite stays green.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
… Talos rollover

PR review flagged that values.yaml and the generated README/CRD still
describe the pre-Talos disk layout. The kubernetes chart used to
provision a separate disk-kubelet PVC for kubelet/containerd state;
the Talos worker boots from a single system disk imaged from the
factory artifact and the chart no longer renders disk-kubelet at all.

- values.yaml: diskSize description now covers the consolidated single
  system disk (Talos OS image + kubelet state + containerd image cache
  + local-path PVCs) instead of the pre-Talos "kubelet and containerd
  data" wording.
- values.yaml: nodeHealthCheck.nodeStartupTimeout no longer hints at
  air-gapped mirrors of factory.talos.dev — Phase 1 has no
  machine.registries.mirrors plumbing, so the hint was promising
  behaviour the chart does not deliver (called out separately in the
  README Breaking Changes section).
- README.md: migration-41 paragraph rewritten so the persistence model
  reflects the single Talos system disk rather than the removed
  disk-kubelet layout.
- Regenerated values.schema.json, api/apps/v1alpha1/kubernetes/types.go,
  and packages/system/kubernetes-rd/cozyrds/kubernetes.yaml via
  `make generate` so the chart, Go API, and tenant CRD all carry the
  same description.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
…efile tagging

PR review flagged that the Dockerfile comment claimed images are mirrored
into ghcr.io/cozystack/cozystack/talos-csr-signer:<commit>, but the
Makefile (image-tags macro) actually tags by chart version
(KUBERNETES_PKG_TAG / IMAGE_TAG) and writes the resulting <digest> into
images/talos-csr-signer.tag. The upstream commit is fingerprinted
indirectly via the digest, not via the tag itself.

Comment-only change in Dockerfile. The images/talos-csr-signer.tag file
itself still pins the upstream ghcr.io/clastix/...:latest digest and
must be regenerated by running `make image-talos-csr-signer` in
packages/apps/kubernetes/ (with REGISTRY=ghcr.io/cozystack/cozystack)
before release — that step needs ghcr.io/cozystack/cozystack push
access, which only the release pipeline / maintainer holds, so the
regeneration is left to the merge handoff rather than baked into
this PR.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
PR review flagged six load-bearing comments that no longer matched
reality after the chart converged on the post-install-hook design and
the cozyrds helm-install-timeout was bumped 15m -> 20m:

- _helpers.tpl: wait-for-kubeconfig deadline rationale referenced the
  15m install timeout. The cozyrds entry now carries 20m
  (release.cozystack.io/helm-install-timeout) — updated, with the
  source of truth (cozyrds annotation) called out so the next reader
  finds the canonical value.
- bootstrap-token-tenant-job.yaml: backoffLimit budget rationale
  likewise referenced 15m; the worst-case ~15m attempt window stays,
  it just now sits well inside the 20m HR window instead of right at
  the edge.
- hack/e2e-apps/run-kubernetes.sh: comment above kubectl wait
  referenced the 15m alignment; corrected to 20m.
- tests/cluster_test.yaml header: described a "$talosReady gate" that
  hid the TalosConfigTemplate document, which no longer exists — the
  TCT is produced at runtime by the talos-reconcile post-install
  hook, not by cluster.yaml. Rewritten to describe the actual document
  order the test asserts on.
- tests/kubelet_reservation_test.yaml header: same $talosReady
  reference + a now-stale note about "happy-path TCT assertions not
  ported"; updated to point at gpu_node_labels_test.yaml as the
  in-repo example of how to assert on the hook's command literal.
- templates/cluster.yaml: a "Hoist all lookups" block described an
  optimisation that no longer applies — the lookups moved into the
  talos-reconcile hook so cluster.yaml has nothing to hoist any more.
  Removed.

No executable code changed; helm unittest stays at 89 PASS.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
…+ new talos templates + MHC + support-matrix

PR review flagged that new code paths landed without tests. This commit
closes the gap:

- pkg/config/config_test.go: table test for the new
  ParseHelmInstallDisableWaitAnnotation parser, mirroring the sibling
  ParseHelmInstallTimeoutAnnotation test that already pinned that
  function's behaviour. cozystack-api reads the annotation on every
  ApplicationDefinition at startup and a typo here silently flips the
  parent kubernetes HR into "wait for child HRs", reproducing the
  Phase 1 install deadlock the disable-wait annotation was added to
  break. 11 sub-cases cover empty / true / True / TRUE / false / False
  / FALSE / mixed-case (invalid) / integer (invalid) / yes (invalid) /
  garbage (invalid).
- tests/talos_templates_test.yaml: renders + key-field assertions on
  the four new talos/* templates (bootstrap-token-tenant-job,
  talos-pki, talos-reconcile-job, talos-secrets) so a future refactor
  breaking ServiceAccount/Role/CRB wiring is caught locally before
  push. Uses documentIndex constants tied to source-file order with
  comments noting the layout.
- tests/mhc_rendering_test.yaml: pins the MHC maxUnhealthy int-vs-
  percent rendering branch — the CAPI admission webhook rejects a
  quoted plain integer ("0") so the chart strips the quote when no
  percent suffix is present, and the test exercises both arms.
- tests/talos_k8s_support_matrix_test.yaml: happy-path coverage for
  the $talosK8sSupportMatrix guard. Failure-path coverage is left as
  a follow-up — values.schema.json's Version enum is strictly tighter
  than the matrix window right now, so the fail() branch is
  unreachable from the test harness until either side relaxes.

Total: 11 new Go subtests + 8 new helm-unittest tests; suite goes 87
-> 97 passing.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
…rovider's pattern

PR review flagged that capi-providers-bootstrap had no update target
documenting how files/bootstrap-components-talos.yaml and
files/components-talos.gz were produced — the cabpt-manager
ServiceAccount rename was hand-applied to a vendored bundle and the
process was undocumented, blocking reproducible bumps of
CABPT_UPSTREAM_VERSION. The sibling capi-providers-cpprovider package
already has a working update target; this commit mirrors that pattern
so both packages have symmetric maintenance flow.

- Makefile: new `update` phony target that downloads the
  CABPT_UPSTREAM_VERSION release artifact from siderolabs/cluster-api-
  bootstrap-provider-talos, applies patches/, writes the result to
  files/bootstrap-components-talos.yaml, and regenerates the gzip
  bundle. CABPT_UPSTREAM_VERSION is the single bump knob; the comment
  block points at the two other call-sites (BootstrapProvider
  spec.version + the clusterctl ConfigMap name) so the next operator
  bumping versions can keep them in lockstep without grepping.
- patches/0001-rename-default-sa-to-cabpt-manager.diff: the
  cozystack-side patch that replaces upstream's default ServiceAccount
  references with cabpt-manager, captured as a unified diff so
  `patch(1)` can re-apply it deterministically on top of any future
  CABPT_UPSTREAM_VERSION. Without the rename, capi-operator rewrites
  the SA namespace (`default` is the SA's literal name in upstream and
  capi-operator treats that as the default SA needing rewrite) and the
  CABPT Deployment loses its RBAC — see PR cozystack#2610 for the original
  diagnosis.
- files/components-talos.gz: regenerated via the new target using
  `gzip -n` so the bundle is byte-reproducible (no embedded mtime or
  filename), making future `make update` runs idempotent. Decompressed
  content is identical to the previous file.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>

@myasnikovdaniil myasnikovdaniil left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

NOT LGTM — every code finding from my earlier reviews is now resolved and verified, but the worker-bootstrap rewrite has never passed E2E: CI fails in the base install on the unrelated kube-controller-manager panic (#2863; fix #2867 is not yet on this branch's base) before the kubernetes app test runs, so the chicken-and-egg fixes are verified only by code reading — on a change that rolls every existing tenant onto Talos.

Business context: Phase 1 of the Kubernetes-app split (cozystack/community#8) — replace the Ubuntu+kubeadm worker path with Talos driven by CABPT + a talos-csr-signer sidecar, existing tenants rolling over automatically.

Prior findings — all resolved ✅

Re-verified against the current head. All 10 inline + 4 body findings from the first review, plus the three follow-up findings, are addressed:

Finding Status Evidence
Silent destructive rollover (v1.30/v1.31 → Talos) Support-matrix fail() guard in cluster.yaml; migration 44 pins KubeadmConfigTemplate with resource-policy=keep (strict, exit-1-retry) so Helm can't prune them mid-rollover; migration 45 bumps v1.30→v1.31 ahead of the guard
Talos version/schematic hardcoded ×3 Hoisted to values.yaml talos.version / talos.schematicID, single source
:latest sidecar + missing resources/securityContext talos-csr-signer built from source, digest-pinned, resources + securityContext added
5×N lookup calls in the nodeGroups loop Hoisted out of the loop, looked up once
Support-matrix guard + trim enum Matrix matches the official Talos v1.13 docs: k8s 1.31–1.36 (verified)
dnsDomain / pod+service CIDRs hardcoded tenantClusterDomain helper + $podCIDR/$serviceCIDR constants
bootstrap-token Job retry budget (~2.5h) backoffLimit: 5, internal retry loop dropped
Hardcoded kubectl image images.kubectl override (digest tag)
cpprovider transitional fork / version skew Rebuilt from upstream v0.19.0 + backport patch (PR #347), image digest-pinned, 0.17–0.19 registered in metadata
maxUnhealthy: "0" rejected by MHC webhook Quotes only percentage values, otherwise renders a bare int — the admission webhook accepts the default
MachineDeployment references a TCT the chart never renders ✅ in code TCT moved to the post-install talos-reconcile hook; the CAPI MachineSet blocks on its absence and unblocks the moment the hook applies it (documented inline)
Hook gated behind addon readiness it must itself unblock ✅ in code New release.cozystack.io/helm-install-disable-wait annotation → Install/Upgrade.DisableWait, so helm-controller no longer blocks the hook on in-tenant addon readiness. Wired end-to-end: ApplicationDefinition annotation → start.go parse → rest.go apply

The three follow-up threads (maxUnhealthy / MachineDeployment→TCT / hook-ordering) are fixed in code but still marked unresolved — please resolve them.

Blocker

B1 — The Talos worker path has no passing E2E; the deadlock fixes are unverified end-to-end

The latest E2E run fails in "Install Cozystack into sandbox" with kube-controller-manager segfaulting on all three nodes — the openapi.isExtension → IsXEmbeddedResource → TypeChecker.Check → validatingadmissionpolicystatus panic tracked in #2863 (fix #2867 is still open and not on this branch's base). This is not caused by this PR — no file here touches that VAP/openapi path, and the only server-side change (start.go) just adds the disable-wait annotation parser. But it dies in the base install, so hack/e2e-apps/run-kubernetes.sh — the only thing that exercises CABPT and Talos workers — never runs.

Impact: the chicken-and-egg fixes (DisableWait + reconcile hook) are proven only by code reading and dev-cluster screenshots, on a PR that rewrites the worker bootstrap for every existing tenant.

Fix: rebase once #2867 lands (or cherry-pick it) to get a green E2E that actually boots a Talos worker through the hook path — or attach manual E2E logs showing a tenant reaching Ready on a Talos worker.

Non-blocking follow-ups

  1. Commit history — please rebase the churn into clean logical commits (not one giant squash). The branch carries a dead revert pair (feat(cpprovider): rebuild kamaji control-plane provider from upstream master + its Revert, superseded by the v0.19.0 rebuild), regenerate-after-rebase noise (README/CRD/deepcopy/cozyrds), abandoned-approach cleanup commits, and ~25 incremental fix(kubernetes): commits patching code introduced earlier in the same PR. The contributing checklist asks for a branch "rebased on upstream/main (no extra commits)". Suggested logical units: CABPT provider · cpprovider rebuild+patch · Talos PKI+sidecar · worker bootstrap+reconcile hook · DisableWait API · migrations · tests · docs/cleanup — and drop the revert pair entirely.
  2. DisableWait applies to Install and Upgrade — the tenant kubernetes HelmRelease now reports Ready as soon as Helm completes, never blocking on actual worker/addon readiness; the safety net shifts entirely to WorkloadMonitor and the hook Jobs' own failure modes. Worth a one-line acknowledgement that any future worker-rollout regression is invisible at the HelmRelease level by design.
  3. Migration 45 ordering — a best-effort failed v1.30→v1.31 bump leaves that tenant on v1.30, and its kubernetes HelmRelease then hits the support-matrix fail() guard on the next reconcile (the annotation surfaces it). Is there a guaranteed ordering that migration 45 completes before the kubernetes chart upgrade reconciles? (Minor: migration 45's top comment says "leaves the version stamp at 44" — should read 45.)
  4. Docs — confirm the website docs get a Talos-worker update (diskSize semantics, the factory.talos.dev prerequisite, air-gapped note). The in-repo README is regenerated and fine.

@lexfrei

Copy link
Copy Markdown
Contributor

Superseded by #2931, which landed the same change on 26 June under an identical title. Closing to keep the queue honest.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/kubernetes Issues or PRs related to the tenant Kubernetes app area/platform Issues or PRs related to platform infrastructure (bundle, flux, talos, installer) do-not-merge/hold Indicates that a PR should not merge because someone has issued /hold kind/feature Categorizes issue or PR as related to a new feature size/XXL This PR changes 1000+ lines, ignoring generated files

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants