Skip to content

[Backport release-1.5] fix(tenant): gate VM-resource HelmReleases on victoria-metrics-operator readiness - #3166

Merged
myasnikovdaniil merged 1 commit into
release-1.5from
backport-3141-to-release-1.5
Jul 3, 2026
Merged

[Backport release-1.5] fix(tenant): gate VM-resource HelmReleases on victoria-metrics-operator readiness#3166
myasnikovdaniil merged 1 commit into
release-1.5from
backport-3141-to-release-1.5

Conversation

@github-actions

@github-actions github-actions Bot commented Jul 1, 2026

Copy link
Copy Markdown

Description

Backport of #3141 to release-1.5.

…rator

The tenant etcd, ingress and monitoring HelmReleases create VM* custom
resources (VMPodScrape directly in etcd/ingress; VMCluster, VMAlert,
VLCluster, VMAlertmanager and VMServiceScrape via the monitoring-system
child of monitoring) guarded by the victoria-metrics-operator validating
webhook with failurePolicy: Fail. On a cold install the operator pod can
be briefly unavailable on :9443, so these releases reconcile before the
webhook backend is serving and are rejected with connection refused. They
carry install.remediation.retries: -1 and would eventually recover, but
the failed-install -> rollback -> retry churn can outlast a cold-install
readiness deadline and surfaces as spurious install failures.

Add a Flux spec.dependsOn from each of these releases to the
victoria-metrics-operator HelmRelease in cozy-victoria-metrics-operator,
so they wait for the operator to be Ready before creating VM* resources.
The system monitoring components gate on the operator the same way, via a
variant-level dependsOn. The cross-shard reference (tenant releases are
sharded, the operator release is not) resolves via helm-controller's
uncached dependency check (APIReader, v1.1.0+).

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
(cherry picked from commit 6db5ae9)
@dosubot dosubot Bot added area/platform Issues or PRs related to platform infrastructure (bundle, flux, talos, installer) backport Should change be backported on previous release kind/bug Categorizes issue or PR as related to a bug labels Jul 1, 2026
@myasnikovdaniil
myasnikovdaniil merged commit 4da2f42 into release-1.5 Jul 3, 2026
5 checks passed
@myasnikovdaniil
myasnikovdaniil deleted the backport-3141-to-release-1.5 branch July 3, 2026 08:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/platform Issues or PRs related to platform infrastructure (bundle, flux, talos, installer) backport Should change be backported on previous release kind/bug Categorizes issue or PR as related to a bug

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants