feat(flux): upgrade to v2.8.0 + chart fixes for strict SSA & kstatus (folds #2612) - #2602

Code Review

This pull request updates the FluxCD version to 2.8.x and introduces an opt-in configuration for the Flux Status Web UI in the fluxcd-operator, which is disabled by default. Feedback suggests explicitly disabling the upstream chart's built-in NetworkPolicy for the Web UI to prevent potential resource conflicts with Cozystack's custom Cilium policies.

Actionable comments posted: 2

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In
`@packages/system/fluxcd-operator/charts/flux-operator/templates/httproute.yaml`:
- Around line 1-25: The HTTPRoute resource (httproute.yaml) is rendered when
.Values.web.httpRoute.enabled is true even if the web server is disabled,
risking routing to a non-existent backend on port 9080; update the top
conditional to require both .Values.web.enabled and
.Values.web.httpRoute.enabled so the HTTPRoute (which references the backend
name via include "flux-operator.fullname" and port 9080) is only emitted when
the web server is enabled and the httpRoute feature is enabled.

In `@packages/system/fluxcd-operator/charts/flux-operator/templates/ingress.yaml`:
- Around line 1-42: The Ingress template currently only guards on
.Values.web.ingress.enabled and conditionally renders pathType, which can
produce invalid networking.k8s.io/v1 manifests; change the top-level conditional
so the block only renders when both .Values.web.enabled and
.Values.web.ingress.enabled are true (e.g., require both in the initial {{- if
... -}}), and ensure each path always emits a pathType (use the existing
.pathType when present otherwise render a default like "ImplementationSpecific")
where the backend references include "flux-operator.fullname" and the port name
"http-web".

---

Nitpick comments:
In
`@packages/system/fluxcd-operator/charts/flux-operator/templates/web-clusterrole.yaml`:
- Around line 16-24: The ClusterRole in web-clusterrole.yaml currently grants
cluster-wide read ("apiGroups":["*"], "resources":["*"],
verbs:["get","list","watch"]) and unrestricted impersonation
("resources":["users","groups"], verbs:["impersonate"]), which is overly
privileged for the web UI; change the template to narrow permissions by
replacing the wildcard apiGroups/resources block (the apiGroups/resources/verbs
entries) with only the specific API groups and resources the web UI actually
needs, remove the impersonate verbs unless impersonation is strictly required,
and if impersonation is required restrict scope by converting to a namespaced
Role or by binding the ClusterRole only to the web serviceAccount (instead of
granting impersonation to broad subjects); also add a short comment in
web-clusterrole.yaml explaining the remaining privileges and a note to operators
about the security implications.

In
`@packages/system/fluxcd-operator/charts/flux-operator/templates/web-standard-roles.yaml`:
- Around line 1-131: Add an opt-in warning to the chart README and release notes
explaining the RBAC blast radius when enabling the web UI: document that
enabling web.enabled (controls used by include "flux-operator.createWebRoles"
and include "flux-operator.createWebRolesAggregation") creates ClusterRoles
flux-web-user and flux-web-admin which grant cluster-wide get/list/watch on
apiGroups:["*"], resources:["*"] (including secrets), and that flux-web-edit is
labeled rbac.authorization.k8s.io/aggregate-to-edit:"true" (so it aggregates
into the built-in edit role), causing edit-role principals to inherit Flux verbs
(reconcile/suspend/resume/download), patch/restart on workloads and pod delete
cluster-wide; also mention the related CVE-2026-23990 and recommend keeping the
web feature opt-in and warning operators to review aggregation before setting
createWebRolesAggregation/createAggregation true.

Actionable comments posted: 1

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@hack/e2e-apps/external-dns.bats`:
- Around line 23-25: The script currently calls "kubectl -n tenant-test wait hr
${name}-system --for=condition=ready" without first ensuring the ${name}-system
HelmRelease exists; add the same existence backstop used for ${name} (a loop
using "kubectl -n tenant-test get hr ${name}-system >/dev/null 2>&1; do sleep 2;
done" or equivalent timeout-backed polling) immediately before the wait for
condition=ready to avoid immediate failures, and apply the identical existence
polling change to the other occurrence around lines 48-50 where ${name}-system
is waited on as well.

---

Nitpick comments:
In `@hack/e2e-apps/vminstance.bats`:
- Around line 34-37: The inline resource cleanup that deletes VMDisk/VMInstance
(the kubectl -n tenant-test delete vmdisks.apps.cozystack.io $name
--ignore-not-found --timeout=3m and the similar delete at lines ~115-116) must
be moved from the end of individual tests into the shared teardown() (or the
framework-level cleanup) so failures don't leak resources; retain the current
pre-test delete as-is, but remove the post-success-only deletes and implement
equivalent deletion logic inside teardown() to unconditionally remove VMInstance
and VMDisk resources for the test name/scope.

NOT LGTM — the three earlier blockers are resolved, but the foundationdb imageType change was hand-edited without running make generate, leaving the generated tree inconsistent and the pre-commit check red.

Business context: Flux v2.7.3 → v2.8.0 (helm-controller v1.5, Helm v4 Server-Side Apply + kstatus), plus chart fixes for fields v2.7 silently dropped and v2.8 SSA now rejects.

Blockers

foundationdb imageType default out of sync — make generate not run (pre-commit red)

File: api/apps/v1alpha1/foundationdb/types.go:42

Issue: The imageType default was switched to split in values.yaml, values.schema.json, README.md and the RD schema, but api/apps/v1alpha1/foundationdb/types.go still carries +kubebuilder:default:="unified", and packages/apps/foundationdb/README.md keeps the table column sized for the old longer value.

Evidence: The pre-commit job fails with exit 123 on an uncommitted make generate diff covering exactly these two files. Reproduced locally with cozyvalues-gen 1.5.0: make generate -C packages/apps/foundationdb rewrites types.go to +kubebuilder:default:="split" and reflows the README table — byte-identical to the CI diff. types.go is not touched anywhere in this PR.

Impact: Runtime CR defaulting is unaffected — the dynamic CRD defaults imageType from the RD openAPISchema, which already says split, so the non-disruptive-upgrade guarantee holds. But the pre-commit check is red (merge-blocking) and the generated source tree is internally inconsistent.

Fix: Run make generate -C packages/apps/foundationdb and fold the result into the imageType commit.

Resolved since the previous review (verified)

• migrations.targetVersion bumped 43 → 44: migration 43 now falls inside seq CURRENT 43; the file exists and the missing-file guard is intact. The migrations image digest is auto-stamped at build time (same convention as migration 42), so leaving it untouched is correct.
• kubevirt-instancetypes: the gn1.* GPU instancetypes are re-added in extra-instancetypes.yaml byte-for-byte identical to the previous catalog; common-instancetypes is pinned to a documented commit; only the EOL centos.7*/centos.stream8* preferences are dropped (release-noted with repoint guidance); both READMEs are regenerated to match.
• The tenant gateway HelmRelease no longer sets upgrade.force; no force: true remains anywhere under packages/ (excluding vendored charts); the new invariant suite asserts notExists on spec.upgrade.force across all six swept HRs and renders each one (non-vacuous).
• rest.go annotation comment corrected; both kubernetes-rd and tenant-rd carry the annotation.
• Test coverage added for the WorkloadMonitor replica-path fix, the enableServiceLinks relocation, the no-upgrade.force invariant, and the new seaweedfs-db chart; the kafka and seaweedfs-db test Makefile targets were wired (previously absent, so the pre-existing kafka suites were not running in CI at all).
• Kubernetes 1.33 minimum: confirmed accurate — Flux v2.8 supports 1.33/1.34/1.35; the release note documents both the platform and the default-off tenant addon.

Non-blocking follow-ups

1. The default-off tenant Flux addon ships Flux v2.8, which requires Kubernetes 1.33+, while tenant clusters can still run v1.30–1.32. This is release-noted but not version-gated; a preflight gate would be more robust than a release note for operators who do not read changelogs.
2. No sibling docs PR was found for the user-facing K8s 1.33 prerequisite bump. The requirements page does not appear to pin a minimum, so there may be nothing stale to fix — worth a glance.

NOT LGTM — two upgrade regressions for existing clusters: migration 43 strips the deletion-protection label off the cozystack-version ConfigMap (undoing migration 42), and dropping the EOL centos.7*/centos.stream8* preferences hard-fails the HelmRelease render of every existing VM that references them, with no migration or compatibility alias.

Business context: Flux v2.7.3 → v2.8.0 (helm-controller v1.5, Helm v4 Server-Side Apply + kstatus), plus chart fixes for fields v2.7 silently dropped and v2.8 SSA now rejects.

Blockers

B1 — Migration 43 strips the no-delete label from the cozystack-version ConfigMap

File: packages/core/platform/images/migrations/migrations/43:30-32

Issue: The version stamp at the end of migration 43 emits a label-less manifest and kubectl applys it under the same client-side field manager migration 42 used:

kubectl create configmap -n cozy-system cozystack-version \
  --from-literal=version=44 --dry-run=client -o yaml | kubectl apply -f-

Migration 42 deliberately applied this ConfigMap with platform.cozystack.io/no-delete: "true", and its own header warns: "a label-less apply by the same field manager would strip it." Migration 43 performs exactly that label-less apply.

Evidence:

• migrations/42 applies the ConfigMap with the no-delete label via heredoc + kubectl apply, with an explicit comment that a later label-less apply by the same manager would strip it.
• templates/cozystack-version.yaml only renders the labeled ConfigMap {{- if not $configMap }} — first install only. On an existing cluster the template renders nothing, so the chart does not restore the label after the migration removes it.
• templates/deletion-protection.yaml defines cozystack-no-delete-guardrail, a ValidatingAdmissionPolicy matching objects with platform.cozystack.io/no-delete: "true". Once the label is stripped, the guardrail no longer covers cozystack-version.
• The upgrade path that runs migration 43 is not exercised by E2E (fresh installs stamp version = targetVersion via the chart and skip the migration hook), so the green E2E run does not validate this.

Impact: Every cluster upgrading through migration 43 loses deletion protection on cozystack-version — the ConfigMap that anchors migration-version tracking. An accidental kubectl delete configmap cozystack-version -n cozy-system then succeeds and can cause migrations to re-run or version detection to break on the next upgrade. This regresses the exact protection migration 42 added.

Fix: Stamp the version with the labeled manifest mirroring migration 42 (include metadata.labels.platform.cozystack.io/no-delete: "true"), and add a test asserting the label survives the stamp. Better: factor the version stamp into a shared helper so future migrations cannot drift from the labeled pattern.

B2 — Removing the EOL centos.7* / centos.stream8* preferences hard-breaks existing VMs on upgrade

File: packages/system/kubevirt-instancetypes/templates/preferences.yaml (removal) + packages/apps/vm-instance/templates/vm.yaml:4-6 (hard fail)

Issue: centos.7, centos.7.desktop, centos.stream8, centos.stream8.desktop, centos.stream8.dpdk are removed from the preference catalog and from the vm-instance schema enum, and are not re-added anywhere. vm.yaml hard-fails the render when a referenced VirtualMachineClusterPreference is absent from the cluster. Any existing VMInstance using one of these profiles fails its HelmRelease render on the next reconcile after upgrade, with no automatic remediation — and the operator cannot patch the object through the API either, because the dynamic CRD enum now rejects the stored value.

Evidence:

• origin/main:.../preferences.yaml ships these five preferences; HEAD ships none of them, and they are absent from extra-instancetypes.yaml.
• vm-instance/values.schema.json drops the five centos.* enum entries.
• vm.yaml:4-6: {{- if and .Values.instanceProfile (not (lookup ... "VirtualMachineClusterPreference" "" .Values.instanceProfile)) }}{{- fail ... }}.
• The PR already solves this exact failure mode for the in-use gn1.* instancetypes by re-adding them as deprecated aliases in extra-instancetypes.yaml; the header comment states the purpose verbatim: "vm.yaml fails the Helm render when a referenced VirtualMachineClusterInstancetype is absent ... appended ... to keep those workloads renderable."

Impact: Hard, unrecoverable HelmRelease render failure for every existing VM on these profiles, on a platform whose target operator does not read changelogs. The "repoint manually before upgrading" note in the PR body is the operator-facing symptom, not a mitigation.

Fix: Re-add the five centos.* preferences as deprecated-alias VirtualMachineClusterPreference objects, mirroring the gn1.* treatment in extra-instancetypes.yaml; or ship a repointing migration (same class as migration 43) that rewrites instanceProfile on existing VMInstances. EOL guest OS is not a reason to hard-fail a running VM's reconciliation — the deprecated-alias mechanism already in this PR keeps such objects renderable while hiding them from new use.

Non-blocking follow-ups

1. Stale Flux version label on the tenant sharded controller. internal/fluxinstall/manifests/fluxcd-tenants.yaml bumps helm-controller to v1.5.0 (Flux 2.8.0) but keeps app.kubernetes.io/version: v2.7.3 on the flux-tenants Deployment (line 8). packages/core/flux-aio/Makefile's update target only syncs the container image into this file via yq; it never touches the version label, so the label drifts on every bump and make update will not self-correct. Cosmetic (the running image is correct) but misreports the Flux version to anything reading the standard label. Fix: update the label, or extend the yq patch to carry app.kubernetes.io/version too.
2. K8s 1.33 minimum is release-noted but not gated. Flux v2.8 requires Kubernetes 1.33+, while tenant clusters can still run v1.30–1.32 and would deploy this Flux if the (default-off) tenant Flux addon is enabled. A preflight version gate would be more robust than a changelog note. No sibling docs PR is open in the website repo.

Resolved since the prior round (verified)

• foundationdb imageType generated-tree drift: 5c4e927 regenerates types.go to +kubebuilder:default:="split" and reflows the README; the "Verify generated code" and pre-commit gates are now green.
• The fd59dea flux-operator/flux-instance chart bump v0.48.0 → v0.50.0 is the operator version only; distribution.version stays pinned at 2.8.x, the chart delta is additive (optional service.ipFamilyPolicy, AWSCodeCommit CRD validations), and the embedded bootstrap installs raw Flux controllers, so there is no operator/bootstrap skew.

LGTM — both blockers from the prior round are resolved and verified; the only open items are the two known non-blocking follow-ups, both acknowledged.

Business context: Flux v2.7.3 → v2.8.0 (helm-controller v1.5, Helm v4 Server-Side Apply + kstatus), plus chart fixes for fields v2.7 silently dropped and v2.8 SSA now rejects.

Resolved since the prior round (verified)

• B1 — migration 43 no longer strips the no-delete label (797284ab). The version stamp now applies cozystack-version from a labeled heredoc carrying platform.cozystack.io/no-delete: "true", mirroring migration 42, so the ConfigMap stays under the cozystack-no-delete-guardrail ValidatingAdmissionPolicy on upgrade. Same client-side field manager as migration 42 → the relabel is a clean 3-way merge, no conflict. The inline comment documents why the label must be carried here.
• B2 — EOL centos.7* / centos.stream8* preferences retained as deprecated aliases (27edd73b). All five (centos.7, centos.7.desktop, centos.stream8, centos.stream8.desktop, centos.stream8.dpdk) are re-added to extra-instancetypes.yaml verbatim from the pre-removal catalog (only an explanatory comment prepended), marked instancetype.kubevirt.io/deprecated: "true" + tags: hidden, mirroring the gn1.* treatment — so vm.yaml's lookup/fail finds them and existing VMs keep rendering. The enum generation now unions preference names from extra-instancetypes.yaml, so values.schema.json and the RD openAPISchema enum both regain the five names and the dynamic CRD no longer rejects the stored values. Evidence: reconstructing instanceProfile.enum from the source catalogs with the Makefile's own yq/jq pipeline reproduces the committed schema byte-for-byte (60 entries); the RD enum matches values.schema.json; there are no object-name collisions between extra-instancetypes.yaml and preferences.yaml/instancetypes.yaml; and the "Verify generated code is up to date" + pre-commit checks are green — so the generated tree is consistent.

Non-blocking follow-ups (carried over, both acknowledged)

1. internal/fluxinstall/manifests/fluxcd-tenants.yaml:8 still reads app.kubernetes.io/version: v2.7.3 while the Deployment now runs helm-controller:v1.5.0 (Flux 2.8.0). Cosmetic (the running image is correct) but misreports the version; make update won't self-correct since the yq patch only syncs the image.
2. K8s 1.33 minimum is release-noted but not gated; a preflight version gate on the default-off tenant Flux addon would be more robust than a changelog note. No sibling website-repo docs PR is open.

No new issues introduced by either commit.

LGTM — the last open item (non-blocking follow-up #1) is now resolved and verified; no regressions, CI green.

Business context: Flux v2.7.3 → v2.8.0 (helm-controller v1.5, Helm v4 Server-Side Apply + kstatus), plus chart fixes for fields v2.7 silently dropped and v2.8 SSA now rejects.

Resolved since the prior round (verified)

• Stale Flux version label synced (910b505d). The flux-tenants sharded-controller Deployment now carries app.kubernetes.io/version: v2.8.0 (was v2.7.3), matching the helm-controller:v1.5.0 it runs. The flux-aio Makefile update target was extended with a yq patch that sources the label from the flux Deployment in fluxcd.yaml, so the label self-corrects on future bumps instead of drifting. Evidence: the source flux Deployment in fluxcd.yaml carries v2.8.0; re-running the new yq expression against the committed tree is a no-op (idempotent); fluxcd-tenants.yaml holds a single document, so the selector-less patch has no collateral; no v2.7.3 references remain under internal/ or packages/.

Non-blocking follow-up (deferred with rationale)

1. K8s 1.33 preflight gate on the default-off tenant Flux addon — deferred to a separate change (it is a new feature; the requirement is already captured in the release note). A tracking issue is the right home for it.