v1.6.1 (2026-08-05)

A patch release with seven fixes covering PostgreSQL, etcd, managed Kubernetes, Keycloak, cozystack-basics, and SeaweedFS, plus release-pipeline reliability fixes and a talm update adding declarative Talos preset knobs.

Fixes

• fix(postgres-operator): align CNPG operator and CRDs to 1.28.2 for PVC resize-deadlock fix: A simultaneous resources + size change on a single-instance PostgreSQL cluster could make the CloudNativePG operator delete the sole primary Pod, classify the PVC as resizing, and never recreate the Pod — wedging the cluster with zero instances and leaving the filesystem resize incomplete. Bumping the operator image and CRDs together to 1.28.2 (which carries upstream's fix, cloudnative-pg#9980 / cloudnative-pg#9981) resolves the deadlock (@scooby87 in #3510, backport #3542).

• chore(etcd-operator): bump etcd-operator to v0.5.4: Rolls up four upstream controller bug fixes: the operator no longer exempts the bootstrap seed from crash-loop self-heal, self-heal now also covers memory-backed etcd members, --initial-cluster-state is derived from cluster phase instead of the seed, and each EtcdCluster's PodDisruptionBudget switches from maxUnavailable to minAvailable (existing clusters are reconciled onto the new field automatically on upgrade). Tenant etcd clusters recover more reliably from member crashes and are less likely to have their PDB block a node drain (@androndo in #3529, backport #3538).

• fix(kubernetes): render the talos-reconcile Job for the default md0 group: On a managed Kubernetes cluster left with the default (undeclared) nodeGroups, the Job that creates each worker's TalosConfigTemplate and patches the control-plane's cert SANs only iterated the user-supplied node-group map, so it silently skipped the implicit md0 group. Nothing failed at install time, since md0 defaults to zero replicas, but the first scale-up (for example, autoscaler-driven growth after enabling ingress-nginx) left new Machines permanently blocked with no matching TalosConfigTemplate. The Job now iterates the same helper that produces the MachineDeployment, so md0 gets its reconcile Job like any explicitly declared group (@myasnikovdaniil in #3535, backport #3536).

• fix(keycloak-configure): patch HelmRelease in release namespace on teardown: The keycloak-configure pre-delete Job cleared the Flux HelmRelease finalizer in a hardcoded namespace that did not match where the release actually installs (cozy-keycloak), so its ServiceAccount was forbidden to patch it, the Job retried forever, and the HelmRelease stuck in Terminating — blocking any uninstall or reinstall of Keycloak. The teardown Job now templates both the release name and namespace from the Helm release itself, so teardown completes correctly (@lexfrei in #3372, backport #3478).

• fix(cozystack-basics): gate the hostname VAP policies on the VAP API: The hostname ValidatingAdmissionPolicy templates rendered unconditionally, so a first install on a cluster where the ValidatingAdmissionPolicy API is unavailable dropped the policies permanently — a later cluster upgrade that gains the API would not bring them back. The templates are now gated on .Capabilities.APIVersions.Has, so they render only where the API exists and are picked up automatically once it becomes available (@lexfrei in #3409, backport #3442).

• fix(seaweedfs): make naming audit fail closed on kubectl and payload errors: hack/seaweedfs-naming-audit.sh, used by operators to classify SeaweedFS instances before the naming-migration cleanup, was fail-open — any kubectl failure or unreadable Helm release payload produced an empty result table indistinguishable from a genuinely clean fleet. Since the runbook uses this script as the gate before deleting old PVCs, a transient API error could have green-lit destroying live data. Every query now fails loudly on error instead of silently reporting "nothing found," and incomplete evidence now falls back to a conservative "direction cannot be established" verdict rather than a wrong deletion candidate (@myasnikovdaniil in #3436, backport #3474).

• chore(release): don't activate kubernetes-nodes on the release-1.6 line: The kubernetes-nodes app package was not ready to ship on the 1.6 line, so its single include is removed from the iaas platform bundle — the platform stops activating the package on this line while its code, API types, and kubernetes-nodes-rd system package are all kept intact for when it is ready (@myasnikovdaniil in #3437).

Development, Testing, and CI/CD

• ci(release): carry the finalize fixes onto the 1.6 line: release-1.6 was cut before three release-pipeline fixes landed on main: dropping persist-credentials on checkout (which had let a stale GITHUB_TOKEN silently win over the app token, so the stable tag push created no workflow run and the automated changelog/docs backstops never fired, as happened for v1.6.0), publishing the GitHub release with the merged changelog as its body instead of a placeholder, and dropping a paths-ignore filter that could drop a changelog-only promotion PR. All three are backported so v1.6.1 releases correctly with real release notes (@myasnikovdaniil in #3530).

• fix(ci): overlay images from the PR base branch, and publish per-line artifacts: PR validation always overlaid unbuilt packages from cozystack-packages:main, so a release-1.6 PR was tested against main's controller binaries against its own line's charts — which is exactly what made #3437 above fail install deterministically with a schema-validation error. Each maintained release-X.Y branch now builds and publishes its own packages artifact, and PR validation overlays from the artifact matching the PR's own base branch (@myasnikovdaniil in #3471, backport #3514).

• fix(release): make promote-retag digest verification media-type-agnostic: The v1.6.0 finalize run aborted partway through promoting rc images to stable tags because its post-copy digest check used skopeo inspect --format '{{.Digest}}', which prints nothing for OCI artifacts like cozystack-packages, leaving most repositories without a stable tag and skipping the installer publish. The digest is now computed as the sha256 of the raw manifest, which works identically for container images and OCI artifacts, so promotion can no longer abort mid-way on this class of artifact (@myasnikovdaniil in #3435, backport #3473).

Other repositories

talm v0.34.0

• [talm] feat(charts): add preset value knobs: Exposes timeServers, control-plane component extraArgs, registryMirrors, per-host registryTLS, multiple Layer2 vips, network.preserveExisting, and network.extraLinks (bonds, VLANs, extra addresses and routes) as values on the cozystack, generic, and talm presets, so a node's Talos machine config can be described declaratively instead of via a template fork. Every knob defaults empty and a stock render stays byte-identical; each input Talos would reject fails fast at render time with a hinted error (@lexfrei in cozystack/talm#232).

• [talm] chore(deps): migrate to Helm 4 and drop the cozystack/talos fork: Moves talm's vendored Helm template engine from v3 to v4 and drops the cozystack/talos fork (carried solely for a --skip-verify flag, now reimplemented locally), tracking stock upstream Talos v1.13.7. Golden render snapshots confirm the generated machine config is unchanged for users (@lexfrei in cozystack/talm#231).

Documentation

• [website] feat(blog): add Cozystack 1.6 release and Blockstor announcement: Publishes the v1.6.0 release-announcement blog post — covering Talos Linux tenant workers, tenant-controlled OIDC, the SecurityGroup API, hierarchical quotas, and in-place etcd-operator adoption — alongside a companion post announcing the open-sourcing of Blockstor, the LINSTOR-compatible storage control plane (@tym83 in cozystack/website#641).

• [website] chore(blog): repair front matter, links and bundle names: Follow-up cleanup on the two new blog posts, fixing front matter, internal links, and page bundle names (@tym83 in cozystack/website@4b5d1ad).

• [website] chore: upgrade Hugo to 0.164.0 and convert HTML content to markdown: Upgrades the site generator to Hugo 0.164.0, requiring Node 22 under its new node permission model, and converts remaining raw-HTML content to markdown along the way, keeping the site buildable on current tooling (@tym83 in cozystack/website#636).

• [website] chore(blog): make the two taxonomy axes disjoint: Cleans up the blog's tagging so its two taxonomy axes no longer overlap, making blog post categorization and filtering more consistent (@tym83 in cozystack/website#635).

• [website] docs(talm): describe the preset value knobs for network and registries: Documents the new talm preset value knobs (cozystack/talm#232) for time servers, control-plane extra args, registry mirrors and TLS, VIPs, and network links, so operators can find the declarative equivalents to a template fork (@lexfrei in cozystack/website#633).

• [website] feat(hack): generate docs from a pre-tag ref and fail loudly on fetch errors: Hardens the docs-generation tooling to build from a ...

Cozystack v1.6.0

Cozystack v1.6.0 replaces the Ubuntu + kubeadm bootstrap of tenant Kubernetes workers with Talos Linux driven by Cluster API, completes the etcd-operator v1alpha2 migration with in-place adoption of live clusters, adds OIDC single sign-on for tenant kube-apiservers and per-instance Grafana, introduces the tenant-facing SecurityGroup firewall API, makes tenant resource quotas hierarchical so a sub-tenant can no longer escalate past its parent's budget, gives Keycloak an optional KMS-encrypting database proxy, a separate admin hostname and S3 backups, propagates operator-provided wildcard certificates down to every tenant termination point, and reworks the release pipeline around immutable tags and rc→stable promotion so a stable release is the byte-identical promotion of the release candidate that was tested. A ten-PR sweep also makes application deletion actually reclaim its storage instead of leaking PVCs. The release rolls up every fix from v1.5.1, v1.5.2 and v1.5.3.

Platform components bumped in this release: Talos v1.13.0 → v1.13.6 (KVM guest-escape CVE fixes), etcd-operator v0.4.5 → v0.5.2 (new etcd-operator.cozystack.io/v1alpha2 API), Cilium 1.19.3 → 1.19.5, KubeVirt v1.8.4, Velero 1.17.0 → 1.18.1, Vertical Pod Autoscaler 1.3.0 → 1.5.0, Harbor 2.14.2 → 2.15.1, Keycloak 26.5.2 → 26.6.3, LINSTOR 1.33.2 → 1.33.3 with linstor-csi v1.11.2, FoundationDB operator v2.13.0 → v2.30.0, HAMi 2.8.1 → 2.9.0, Percona MongoDB operator 1.21.1 → 1.22.0, OpenBao v2.5.0 → v2.5.1, CoreDNS chart 1.43.2 → 1.46.0, csi-driver-nfs 4.11.0 → 4.13.3, OpenCost 1.111.0 → 1.120.3, NATS 2.11.8 → 2.11.10, linstor-scheduler v0.3.2 → v0.3.6, and ouroboros 0.8.0 → 0.8.1.

Note: Items marked (shipped in v1.5.x) were also released in the v1.5.1, v1.5.2 or v1.5.3 patch releases.

⚠️ Breaking Changes and Required Actions

This is the largest upgrade surface since v1.0. Read this section in full before upgrading — three of the items below can block or wedge the upgrade if their preconditions are not met.

The platform migration targetVersion moves from 45 (v1.5.x) to 54, which means migrations 45 through 53 run as pre-upgrade hooks on the way to v1.6.0.

Pre-upgrade checks

Run these against the management cluster before applying the v1.6.0 Platform Package.

1. etcd adoption needs a reachable backup target (migration 50 — hard-fails the upgrade).

Migration 50 adopts every legacy etcd.aenix.io/v1alpha1 cluster onto the new v1alpha2 operator and takes a mandatory pre-adoption etcd snapshot first. If it cannot resolve the platform-managed snapshot target it exits 1 and halts the upgrade (#2859, #3270, #3335).

# Do you have legacy etcd clusters at all? If this prints nothing, migration 50 is a no-op.
kubectl get etcdclusters.etcd.aenix.io -A

# The snapshot target: the cozy-default-etcd strategy and the projected credentials Secret.
kubectl get etcds.strategy.backups.cozystack.io cozy-default-etcd
kubectl get secret cozy-backups-creds -n cozy-velero \
  -o jsonpath='{.data.AWS_ACCESS_KEY_ID}{"\n"}'

# The bucket behind it must be healthy.
kubectl get buckets.apps.cozystack.io cozy-backups -n tenant-root

If the Secret or the strategy is missing and you have legacy etcd clusters, either repair the cozy-backups Bucket / backupstrategy-controller first, or accept adopting live etcd without a snapshot by taking the documented escape hatch (see Manual actions required).

2. SeaweedFS naming audit (chart refuses to render for two classes of tenant).

The v1.5.0 SeaweedFS chart bump to 4.31 renamed workloads from chart-based (seaweedfs-*) to release-based (seaweedfs-system-*) names. v1.6.0 pins fullnameOverride: seaweedfs and adopts the running set in place, but two states cannot be adopted automatically and the chart fails the render rather than guess (#3282, #3339).

# Step 0 — seaweedfs-db ownership. A missing row means the filer metadata is ALREADY lost.
kubectl get cluster.postgresql.cnpg.io -A \
  -o custom-columns='NS:.metadata.namespace,NAME:.metadata.name,OWNER:.metadata.annotations.meta\.helm\.sh/release-name,KEEP:.metadata.annotations.helm\.sh/resource-policy'

# Step 1 — classify every SeaweedFS instance (read-only).
hack/seaweedfs-naming-audit.sh

Class L needs no action. Class S (installed fresh on 1.5.x) and class MIXED (both naming generations present) must be recovered before upgrading — follow docs/operations/seaweedfs-431-rename-recovery.md. A cluster that went 1.4.x → 1.6.0 directly never renamed and is unaffected.

3. Tenant Kubernetes clusters still on v1.30.

v1.30 leaves the Talos↔Kubernetes support matrix and the chart now refuses to render it. Migration 46 patches live CRs from v1.30 to v1.31, but a GitOps-managed CR is re-overwritten by the next source reconcile (#2931).

kubectl get kuberneteses.apps.cozystack.io -A \
  -o custom-columns='NS:.metadata.namespace,NAME:.metadata.name,VERSION:.spec.version'

Any row showing v1.30 that is reconciled from Git must have spec.version bumped to v1.31 (or newer) in Git, before or together with the platform upgrade.

4. Hand-made tenant StorageClasses that collide with propagated LINSTOR classes.

Remote-accessible LINSTOR StorageClasses are now created inside each tenant cluster under the same name. A manually created tenant StorageClass with a colliding name (typically replicated) blocks the propagated class and stalls the tenant CSI release (#2872).

# Inside each tenant cluster:
kubectl get storageclass replicated -o jsonpath='{.metadata.annotations.meta\.helm\.sh/release-name}{"\n"}'

Delete any such class that is not Helm-managed before upgrading. Infra classes that must stay node-local need an explicit linstor.csi.linbit.com/allowRemoteVolumeAccess: "false" — an absent annotation is treated as remote-accessible.

5. Deprecated etcd tenant-module backup values.

The backup.* block on the etcd tenant module is removed in this release. Confirm nothing still relies on it (#2859).

kubectl get etcds.apps.cozystack.io -A -o jsonpath='{range .items[*]}{.metadata.namespace}/{.metadata.name}{"\t"}{.spec.backup.enabled}{"\n"}{end}'

6. If you are coming from v1.4.x, the v1.5.0 requirement still applies: the management cluster (and any tenant cluster enabling the Flux addon) must be on Kubernetes 1.33+.

Breaking changes

• Tenant Kubernetes workers move from Ubuntu + kubeadm to Talos Linux. What changed: worker bootstrap is now driven by cluster-api-bootstrap-provider-talos (CABPT) with a talos-csr-signer sidecar in the Kamaji control-plane pod; workers boot the Talos openstack image streamed in by CDI. Who is affected: every existing tenant Kubernetes cluster. What to do: nothing mandatory — existing machines roll over to Talos workers automatically on the first reconcile after upgrade — but plan for a full worker-pool replacement per tenant cluster, and expect worker disks to be reprovisioned and container images re-pulled. The separate disk-kubelet PVC is gone; nodeGroups[*].diskSize now sizes the single Talos system disk (@myasnikovdaniil in #2931).

• Kubernetes v1.30 removed from the tenant support matrix. What changed: the version enum drops v1.30; the chart fails the render with an explicit guard. Who is affected: tenants still pinned to v1.30. What to do: migration 46 bumps live CRs to v1.31; GitOps-managed CRs must be bumped in Git (see pre-upgrade check 3) (#2931).

• Worker MachineHealthCheck remediation is now ON by default. What changed: the worker MHC maxUnhealthy moved from a hard-coded 0 (remediation effectively disabled) to nodeHealthCheck.maxUnhealthy, defaulting to "50%" — CAPI now deletes and replaces unhealthy worker Machines. Who is affected: every tenant Kubernetes cluster. What to do: set nodeHealthCheck.maxUnhealthy: "0%" to keep the previous behaviour until your fleet is stable on Talos workers (#2931).

• The default md0 node group is no longer merged into every cluster. What changed: nodeGroups defaults to {} and the built-in md0 is applied only when no node groups are configured, so it can finally be removed or replaced. Who is affected: clusters that defined their own node groups and were silently also getting md0. What to do: nothing — migration 47 pins md0 explicitly on existing CRs to preserve the live topology; remove it afterwards if you do not want it. Migration 47 is fail-closed: a read or patch failure aborts the upgrade rather than let Helm prune a live md0 MachineDeployment (@myasnikovdaniil in #2936).

• Fresh tenant clusters with nodeGroups: {} come up with zero workers. What changed: the chart no longer manages MachineDeployment.spec.replicas — the cluster-autoscaler owns it alone, seeded from minReplicas: 0. Who is affected: new clusters that enable addons.ingressNginx.enabled: true on a default CR. What to do: either supply a node group with roles: [ingress-nginx] and minReplicas >= 1, or let the autoscaler bring up md0 once the ingress-nginx pods go Pending. Upside for existing clusters: helm upgrade no longer drains workers back to a hardcoded replicas: 2 on every platform bump (@IvanHunters in #3233).

• etcd moves to etcd-operator.cozystack.io/v1alpha2. What changed: the legacy etcd.aenix.io/v1alpha1 StatefulSet model is replaced by a Membership-API lifecycle; the vendored upstream chart is replaced by a Cozystack-authored one at appVersion v0.5.2, and CRDs move ...

v1.5.2 (2026-07-03)

A patch release with ten bug fixes covering managed Kubernetes, MariaDB, Kafka, Harbor, SeaweedFS, VictoriaMetrics-operator dependency gating, Kamaji, and platform install ordering, plus new documentation and a talm update adding encrypted user values.

Fixes

• fix(kamaji): backport datastore unused-deletion deadlock fix: Deleting a Tenant whose etcd DataStore had no more referencing TenantControlPlanes left the DataStore stuck Terminating (blocked by the kamaji.clastix.io/TenantControlPlane finalizer), hanging the etcd Helm uninstall and wedging the tenant namespace. Backports the upstream fix (clastix/kamaji#1122, fixing clastix/kamaji#1115) as a source patch on the pinned kamaji build, so deleting the last control plane on a DataStore now cleans up correctly (@myasnikovdaniil in #3146, backport #3187).

• fix(kubernetes): gate vmop cert-manager-crds dependency on certManager.enabled: The victoria-metrics-operator HelmRelease listed cert-manager-crds in dependsOn unconditionally, even though that HelmRelease is only created when certManager.enabled is true. Clusters running monitoringAgents.enabled=true with certManager.enabled=false had victoria-metrics-operator permanently blocked on a dependency that would never exist, cascading to monitoring-agents and vertical-pod-autoscaler never becoming ready. The dependency is now gated on the same condition that creates it (@myasnikovdaniil in #3130, backport #3185).

• fix(tenant): gate VM-resource HelmReleases on victoria-metrics-operator readiness: The tenant etcd, ingress, and monitoring HelmReleases create VictoriaMetrics-operator custom resources guarded by a validating webhook with failurePolicy: Fail. On a cold install the operator pod could be briefly unavailable, so these releases would reconcile before the webhook was serving and fail with "connection refused" — the failed-install/rollback/retry churn could then outlast the install-readiness deadline and surface as a spurious failure. These releases now dependsOn the victoria-metrics-operator HelmRelease so they wait for it to become Ready first, eliminating the race on fresh installs (@lexfrei in #3141, backport #3166).

• fix(mariadb): re-guard replication so single-replica passes the operator webhook: A prior change made replication.enabled: true unconditional so the operator would always create -primary/-secondary services, but the mariadb-operator validating webhook rejects replication when spec.replicas == 1, so single-replica MariaDB instances could no longer be created — the HelmRelease never became Ready. Replication is now re-guarded on replicas > 1; single-replica installs keep external access via a LoadBalancer service on the bare release name, and backups, RBAC, and dashboards are updated to recognize both topologies (@scooby87 in #3180, backport #3183).

• fix(kubernetes): make node group resources optional when instanceType is set: A managed-Kubernetes node group sized purely by instanceType (with resources omitted) failed values.schema.json validation because resources was required on every node group. resources is now optional: a node group sized purely by instanceType passes validation and installs, while a node group setting both instanceType and explicit resources has the explicit resources take precedence, so KubeVirt does not reject a VM referencing an instancetype while overriding domain.cpu/memory (@lexfrei in #3121, backport #3156).

• fix(seaweedfs): enforce read-only bucket access by bumping cosi-driver to v0.3.1: The bundled SeaweedFS COSI driver (v0.1.2) hardcoded read-write S3 permissions for every BucketAccess and ignored the chart's -readonly BucketAccessClass, so credentials meant to be read-only could still upload and delete objects. Bumping the driver to v0.3.1 makes it honor accessPolicy: readonly and issue Read/List-only credentials, closing the privilege gap for tenants relying on read-only bucket access (@lexfrei in #3136, backport #3153).

• fix(platform): order backupstrategy-controller after cozystack-basics: backupstrategy-controller creates the cozy-backups Bucket in the tenant-root namespace, which cozystack-basics provisions. On a fresh install the Bucket could race namespace creation and fail with namespaces "tenant-root" not found, which then raced the install-wait deadline and failed unrelated fresh installs. A dependsOn edge on cozystack-basics now guarantees the target namespace exists first (@lexfrei in #3037, backport #3079).

• fix(kafka): set entity-operator resources to avoid OOM under tenant LimitRange: Kafka's entityOperator topic-operator and user-operator containers carried no resource requests/limits, so in tenant namespaces — where the default LimitRange caps containers at 128Mi — the JVM processes were OOMKilled on startup. The entity-operator stayed in CrashLoopBackOff, KafkaTopic/KafkaUser objects never reconciled, and topics could get stuck on the strimzi.io/topic-operator finalizer during teardown. Explicit 256Mi/512Mi requests/limits are now set so the operators no longer inherit the namespace default (@myasnikovdaniil in #2934, backport #3054).

• fix(harbor): preserve jobservice and trivy storageClass on upgrade: The cozy-harbor wrapper never forwarded storageClass to the jobservice jobLog PVC, so it was created without a storageClassName and picked up the cluster default via default-StorageClass admission — and because PVC storageClassName is immutable, this produced rendered-vs-live drift on every upgrade. The Trivy StatefulSet's volume claim template had the same latent issue. Both now preserve an existing PVC's class via lookup and fall back to the configured storageClass for fresh installs (@myasnikovdaniil in #2930, backport #3147).

• fix(seaweedfs): point S3 consumers at the reachable S3 service: Renaming the SeaweedFS S3 Service to <name>-s3 left several consumers — the S3 ingress backend, the iceberg ingress backend, and the in-cluster COSI provisioner endpoint — still resolving the old <fullname>-s3 name or the wrong port, so the external S3 endpoint returned 503 via ingress-nginx and in-cluster COSI paths defaulted to port 443 (or the filer service, which serves no S3) instead of the real S3 service on port 8333. All consumers now resolve seaweedfs-s3 on the correct port (@lexfrei in #3119, backport #3127).

Other repositories

talm v0.32.0

• [talm] feat: first-class encrypted user values, honored at template and apply: Adds support for encrypted user secret values (a registry password, a KMS plugin's secret-id, etc.) that chart templates consume — encrypted at rest in git with the existing talm.key, decrypted in memory only at render time, and kept out of committed node files and terminal output. A value file named *.encrypted.yaml referenced from templateOptions.valueFiles is age-decrypted in memory at both talm template and talm apply; talm init --encrypt/--decrypt covers values-secret.yaml ↔ values-secret.encrypted.yaml. talm apply now also honors the full value-source flag set and Chart.yaml's templateOptions, and secret-bearing fields are redacted by default from rendered node files, stdout, and the apply drift preview, with --show-secrets/--show-secrets-in-drift to reveal them on demand (@lexfrei in cozystack/talm#222).

Documentation

• [website] docs(operations): add containerized GPU workloads guide: Publishes a new operations guide for the container variant of cozystack.gpu-operator, covering containerized GPU workloads (CUDA pods, ML training, inference) on GPU nodes that already ship the NVIDIA driver and nvidia-container-toolkit via the distro package manager (@lexfrei in cozystack/website#555).

• [website] docs(talos): warn against changing op: on the machine files block: Adds a warning next to the Talos machine.files block in the bare-metal bootstrap docs, explaining that Talos rejects op: create for files outside /var, so the /etc/lvm/lvm.conf entry must keep op: overwrite — changing it silently fails the WriteUserFiles boot step and reboot-loops the node with only the opaque bootstrap is not available yet error (@lexfrei in cozystack/website#575).

• [website] docs(talm): document chart drift detection and strict mode: Documents the chart drift detection shipping with talm (cozystack/talm#216) — the library-content and preset-baseline drift warnings, the .talm-preset.lock file and the advice to commit it, talm init --update re-sync semantics, and the strictCharts: true/--strict-charts escalation with its unverifiable-baseline behavior (@lexfrei in cozystack/website#577).

• [website] docs(platform): document publishing.certificates wildcard options: Documents two previously-undocumented publishing.certificates.* platform-package options, including the opt-in wildcard mode that issues one shared *.<root-host> certificate via a DNS-01 solver instead of a per-host ACME certificate, helping operators avoid Let's Encrypt rate limits at scale (@lexfrei in cozystack/website#588).

• [website] fix(docs): correct stale version pins and prevent next.yaml drift: The version-pin data file for the v1.5 docs was a stale copy of `n...

v1.4.5 (2026-07-03)

A patch release with fixes for tenant control-plane deletion, VM node-group sizing, Harbor storage-class drift, CSI hot-plug volume handling, and a new talm v0.32.0 with encrypted user values.

Fixes

• fix(kamaji): backport datastore unused-deletion deadlock fix (clastix/kamaji#1122): Deleting a tenant whose etcd DataStore had zero referencing TenantControlPlanes left the DataStore stuck in Terminating (blocked by the kamaji.clastix.io/TenantControlPlane finalizer), which hung the etcd Helm uninstall and wedged the whole tenant namespace. Tenant deletion involving an otherwise-idle DataStore now completes cleanly instead of hanging indefinitely (@myasnikovdaniil in #3146, backport #3188).
• fix(kubernetes): gate vmop cert-manager-crds dependency on certManager.enabled: The victoria-metrics-operator HelmRelease unconditionally depended on cert-manager-crds, even though that HelmRelease is only created when certManager.enabled is true. Clusters with monitoringAgents.enabled=true and certManager.enabled=false therefore failed with a "not found" error on vmop, which cascaded into monitoring-agents and vertical-pod-autoscaler never installing. This previously-broken combination now installs correctly (@myasnikovdaniil in #3130, backport #3186).
• fix(kubernetes): make node group resources optional and prefer them over instanceType: Node groups sized purely by instanceType (with resources omitted) failed values.schema.json validation, even though sizing was fully specified. resources is now optional, and when both instanceType and explicit resources (both cpu and memory) are set, the explicit resources take precedence over the instance type. A partial spec with only one of cpu/memory alongside an instanceType now fails fast at render time with a clear error instead of producing an invalid VM spec (@lexfrei in #3121, backport #3157).
• fix(harbor): preserve jobservice and trivy storageClass on upgrade: The cozy-harbor wrapper never forwarded storageClass to the jobservice job-log PVC, so it was created without a storageClassName, and the default-StorageClass admission then stamped the cluster default onto it. Because storageClassName is immutable, this caused permanent drift between the rendered and live state on every upgrade. The same latent drift existed for Trivy's volumeClaimTemplate. Both now preserve an existing PVC/VCT's storage class via lookup, falling back to the configured storageClass only for fresh installs, so upgrades no longer show spurious storage-class drift (@myasnikovdaniil in #2930, backport #3151).
• fix(kubernetes): add spec.timeout to tenant CSI HelmRelease: The tenant kubevirt-csi-node HelmRelease was the only tenant addon without an explicit timeout: 10m (all others already had it), so it fell back to Helm's default 5-minute timeout. On slower tenant control-plane bring-up, the CSI DaemonSet could still be InProgress past 5 minutes, causing helm-controller to trigger a remediation uninstall/reinstall cycle mid-flight and wedge any in-progress volume mount. Aligning the timeout with the other tenant HelmReleases removes this source of intermittent CSI install/remediation flapping (@myasnikovdaniil in #2727, backport #2929).
• fix(csi): detach orphan hot-plug volumes from VMI when parent VM is gone: Upstream treated a missing parent VM as detach success, so when a VMI outlived its VM during node reprovisioning, the hot-plug pod kept the infra device exclusively attached to the source host and subsequent attach attempts failed. The CSI driver now re-checks VMI volume status after the upstream unpublish call and detaches directly from the VMI when needed, and kubevirt-csi-driver is bumped to the latest upstream release carrying this fix, resolving stuck volume attachments after node reprovisioning (@kvaps in #2866, backport #2965).

Other repositories

talm v0.32.0

• [talm] feat: first-class encrypted user values, honored at template and apply: Adds support for arbitrary encrypted user secret values (registry passwords, KMS plugin secret IDs, etc.) consumed by chart templates. Value files named *.encrypted.yaml are age-decrypted in memory via the existing talm.key at both template and apply time — plaintext never touches disk. talm init --encrypt/--decrypt now covers values-secret.yaml ↔ values-secret.encrypted.yaml. Secret-bearing fields are omitted from committed nodes/*.yaml files and redacted by default in talm template stdout and talm apply's drift preview (--show-secrets/--show-secrets-in-drift reveal them on demand). talm apply also now honors the full set of value-source flags and templateOptions from Chart.yaml, so a value supplied at template time is no longer silently dropped at apply (@lexfrei in cozystack/talm#222).

Documentation

• [website] docs(operations): add containerized GPU workloads guide: Adds a guide covering running GPU workloads in containers on Cozystack, helping operators evaluate and configure GPU-accelerated container deployments (@lexfrei in cozystack/website#555).
• [website] docs(talm): document chart drift detection and strict mode: Documents talm's chart drift detection and strict mode, explaining how operators can catch and control configuration drift between rendered and applied charts (@lexfrei in cozystack/website#577).
• [website] docs(talos): warn against changing op: on the machine files block: Adds a warning against changing the op: field on the Talos machine files block, helping operators avoid a known misconfiguration pitfall (@lexfrei in cozystack/website#575).
• [website] docs(platform): document publishing.certificates wildcard options: Documents the publishing.certificates wildcard options for the platform chart, clarifying how operators can configure wildcard certificates for published services (@lexfrei in cozystack/website#588).

Contributors

Thanks to everyone who contributed to this patch release:

• @kvaps
• @lexfrei
• @myasnikovdaniil

Full Changelog: v1.4.4...v1.4.5

v1.5.1 (2026-06-23)

A patch release that fixes a v1.5.0 regression where Windows VM preferences with persistent EFI/TPM state blocked live-migration and node drains.

Fixes

• fix(kubevirt-instancetypes): drop persistent EFI/TPM to unblock live-migration: v1.5.0 restored persistent EFI/TPM state for the windows.11, windows.2k22, and windows.2k25 KubeVirt instance-type preferences. With persistence enabled, KubeVirt provisions a persistent-state-for-<vm> backend-storage PVC on the default replicated StorageClass, which is ReadWriteOnce — this pins the VM to its current node and blocks live-migration and node drains. On clusters using evictionStrategy: LiveMigrate, the pin can stall cluster upgrades entirely. This fix strips persistent EFI/TPM from the affected preferences using yq del (the previous sed-based strip produced null, which the KubeVirt v1.8 CRD rejects). Secure Boot and the vTPM remain present and functional — Windows VMs still boot normally; EFI/TPM state is simply not persisted across reboots. This is a stopgap until vm-state can request RWX storage (@kvaps in #3006, backport #3016).

Contributors

Thanks to everyone who contributed to this patch release:

• @kvaps

Full Changelog: v1.5.0...v1.5.1

Cozystack v1.5.0

Cozystack v1.5.0 brings Gateway API support via Cilium as an opt-in ingress layer alongside ingress-nginx, TLS for managed databases and messaging (Kafka, NATS, Qdrant, and PostgreSQL external endpoints), backups that work out of the box with a platform-managed default BackupClass, a shared backups bucket, a new etcd backup strategy and a generic Job strategy, the Flux v2.8 upgrade with strict server-side apply and kstatus health checking, a new flux-shard-operator that spreads tenant HelmReleases across helm-controller shards so one noisy tenant can no longer stall the others, operator-provided wildcard certificates for platform and root-tenant ingress, GPU passthrough that works without manual KubeVirt patching, a deletion-protection guardrail for critical platform objects, and runtime-populated dashboard dropdowns via a new Option API. The release also rolls up every fix from v1.4.1 through v1.4.4.

Platform components bumped in this release: Flux v2.7.3 → v2.8.0 (flux-operator/flux-instance charts v0.33.0 → v0.50.0), MetalLB v0.15.2 → v0.16.1 (FRR-K8s is now the default BGP backend), SeaweedFS 4.05 → 4.31, etcd-operator v0.4.3 → v0.4.5, ouroboros v0.7.2 → v0.8.0, seaweedfs-cosi-driver v0.3.1, and the new kuberture system package.

Note: Items marked (backported to v1.4.x) were also shipped in the v1.4.1, v1.4.2, v1.4.3, or v1.4.4 patch releases.

Feature Highlights

Gateway API Support via Cilium

Cozystack-native services can now be exposed through the Gateway API backed by Cilium, as an opt-in alternative to the existing per-tenant ingress-nginx controllers. The feature is materialized per tenant through a new gateway.cozystack.io/v1alpha1 TenantGateway CRD reconciled by cozystack-controller.

Enable it at the platform level with publishing.gateway.enabled=true, then either give a tenant its own Gateway, LoadBalancer IP, and certificate with tenant.spec.gateway=true, or leave it unset and let the tenant inherit the nearest ancestor's Gateway through the same label-based selector model that already drives ingress inheritance. Two certificate solver modes are supported: HTTP-01 (the default — a per-app certificate with zero platform configuration for new apps) and DNS-01 (opt-in — a single wildcard certificate covering an apex, with cloudflare, route53, digitalocean, and rfc2136 providers).

Defaults stay on ingress-nginx, so existing clusters are unchanged. Two things to be aware of: Cilium Envoy / Gateway API is now always enabled (an extra cilium-envoy DaemonSet, roughly 100 MB RAM per node at idle), and cozystack-api now invokes admission (createValidation / deleteValidation) on Create and Delete for apps.cozystack.io/* — so any custom ValidatingAdmissionPolicies or webhooks on those kinds will now fire on all three verbs. See the Gateway API guide (@lexfrei in #2470).

TLS for Managed Databases and Messaging

Four managed-app charts gain TLS support driven by a single tls.enabled value with consistent tri-state semantics: when unset it inherits external (TLS auto-on when the service is published externally, off when cluster-internal), and an explicit true/false always wins. In every case the trust anchor is a chart- or operator-managed self-signed CA that clients retrieve and pin — there is no publicly trusted CA. The one upgrade-time behaviour change to plan for: existing instances with external: true flip to TLS-on after upgrade; cluster-internal instances are unaffected.

• Kafka serves TLS on its external LoadBalancer listener (port 9094), with certificates managed end-to-end by the Strimzi operator; clients trust via the operator-published <release>-cluster-ca-cert / <release>-clients-ca-cert secrets. The external listener is now gated only on external: true, decoupled from tls.enabled (@Arsolitt in #2681).
• NATS and Qdrant gain TLS via a self-contained cert-manager chain (self-signed Issuer → CA → leaf) rendered in the tenant namespace; NATS covers both client connections and cluster routes, Qdrant covers REST and gRPC. Clients trust the <release>-ca secret (@Arsolitt in #2684, #2685).
• PostgreSQL — CNPG already serves TLS unconditionally, so tls.enabled here injects the external hostname into the operator-managed server certificate's SANs (when external: true), so sslmode=verify-full works against the external endpoint. Clients retrieve ca.crt from the <release>-credentials secret (@Arsolitt in #2686).

Backups That Work Out of the Box

This release closes the gap between "backup machinery is installed" and "backups actually work without per-app S3 configuration."

A platform-managed default BackupClass (cozy-default) is now shipped, backed by a system bucket (cozy-backups). Apps opt in with a useSystemBucket flag, after which the platform projects shared backup credentials into the tenant namespace (with RBAC isolation and projection metrics) and skips per-release credential Secrets. Default strategies are provided for every backup-capable app — Velero for VMDisk/VMInstance, CNPG for PostgreSQL, MariaDB, Altinity for ClickHouse, FoundationDB, and etcd — and a Velero BackupStorageLocation is wired to the system bucket. The legacy per-tenant S3 fields on Postgres and ClickHouse are deprecated in favour of this default flow (@androndo in #2716).

To make that default flow reliable, Velero is now a default system package rather than optional. This fixes a deterministic failure where the default backupstrategy-controller (which hard-depends on Velero) sat in DependenciesNotReady and kept the platform HelmRelease from ever reaching Ready. Existing clusters get Velero in the cozy-velero namespace on upgrade; opt out via bundles.disabledPackages if you do not back up VMs (@myasnikovdaniil in #2833).

Two new backup strategies join the catalog: an etcd strategy (cluster-scoped strategy.backups.cozystack.io Etcd CRD, S3-only, with snapshot BackupJob and a destructive in-place RestoreJob), and a generic application-agnostic Job strategy where the operator supplies a Kubernetes Job template that Cozystack renders and runs as a one-shot backup, then re-renders with .Mode == "restore" for recovery — the generic counterpart of the app-specific drivers (@androndo in #2641, @lllamnyp in #1721).

Flux v2.8 Upgrade with Strict Server-Side Apply

Flux is upgraded from v2.7.3 to v2.8.0 across both the embedded management-cluster Flux and the optional tenant Flux addon (flux-operator / flux-instance charts move v0.33.0 → v0.50.0). Flux v2.8's helm-controller v1.5 ships Server-Side Apply with --force-conflicts and kstatus-based health checking by default — so misplaced chart fields that v2.7 silently dropped are now hard errors (fixed here for foundationdb, kafka, kubevirt-instancetypes, vm-instance, and the platform chart), and parent HelmReleases now wait for every child resource to be Ready before reporting Ready themselves.

Action required on upgrade: Kubernetes 1.33+ is now required for the management cluster (and for any tenant cluster enabling the Flux addon). The upgrade.force: true knob is removed, so immutable-field changes (for example StatefulSet volumeClaimTemplates / serviceName) no longer self-heal and must be recreated manually (kubectl delete sts <name> --cascade=orphan). Persistent TPM/EFI is re-enabled for Windows KubeVirt preferences (each affected VM provisions one extra RWO PVC), and FoundationDB imageType is pinned to split to keep upgrades non-disruptive (@myasnikovdaniil in #2602).

flux-shard-operator: Tenant helm-controller Sharding

A new flux-shard-operator spreads tenant HelmReleases across multiple helm-controller shards, so one noisy tenant — for example a HelmRelease stuck in infinite remediation — can no longer degrade reconciliation for everyone else. Placement is per-tenant (all of a tenant's HelmReleases share one shard), assigned greedily by least load, with a CREATE-time mutating webhook stamping the shard label on each HelmRelease.

It ships with shardCount: auto by default, which sizes shards from the tenant HelmRelease count: small clusters stay at a single shard (today's behaviour) while large fleets shard out automatically, and an integer pins the count explicitly. The legacy hand-rolled flux-tenants deployment is drained and retired automatically by migration 44 (@kvaps in #2821).

Operator-Provided Wildcard Certificates

Operators can now serve platform services and the root tenant's ingress under a pre-existing wildcard TLS certificate instead of minting per-host ACME certificates. Set publishing.certificates.wildcardSecretName to the name of a TLS Secret already created in the publishing namespace (tenant-root by default) — only the Secret name travels over the cozystack-values channel, never the key material.

It works on both ingress paths: with ingress-nginx the controller serves it as --default-ssl-certificate and platform Ingresses drop their cert-manager annotations; with Gateway API a new existingSecret TenantGateway cert mode references the Secret directly and provisions no Issuer or Certificate. Scope is the root tenant only for now; extending wildcard mode to child tenants is a follow-up (@lexfrei in #2819).

GPU Passthrough Out of the Box

GPU enablement is wired up a...