Skip to content

AMConfig validation doesn't consider the global AM config #8006

@bogdant-ad

Description

@bogdant-ad

Is there an existing issue for this?

  • I have searched the existing issues

What happened?

Description

The operator fails AM config validation with "mute time interval x not found" even if that mute time interval is defined in the global config. Same applies to other properties requiring definitions.

Steps to Reproduce

  1. Create a global AM config with a mute time interval
  2. Create another AM config referencing the mute time interval
  3. Wait for prom-operator to reconcile them

Expected Result

Validation is passed successfully, since the resulting routing tree has all referenced definitions.

Actual Result

Validation fails with "mute time interval x not found"

Prometheus Operator Version

v0.76.1

Kubernetes Version

clientVersion:
  buildDate: "2025-09-09T19:44:50Z"
  compiler: gc
  gitCommit: 93248f9ae092f571eb870b7664c534bfc7d00f03
  gitTreeState: clean
  gitVersion: v1.34.1
  goVersion: go1.24.6
  major: "1"
  minor: "34"
  platform: darwin/arm64
kustomizeVersion: v5.7.1
serverVersion:
  buildDate: "2024-12-16T21:30:31Z"
  compiler: gc
  gitCommit: 354eac776046f4268e9989b21f8d1bba06033379
  gitTreeState: clean
  gitVersion: v1.30.8
  goVersion: go1.22.9
  major: "1"
  minor: "30"
  platform: linux/arm64

Kubernetes Cluster Type

kind

How did you deploy Prometheus-Operator?

prometheus-operator/kube-prometheus

Manifests

prometheus-operator log output

ts=2025-10-09T08:40:20.658768055Z level=warn caller=/home/runner/work/prometheus-operator/prometheus-operator/pkg/alertmanager/operator.go:1044 msg="skipping alertmanagerconfig" component=alertmanager-controller error="route[3]: mute time interval \"public_holidays\" not found" alertmanagerconfig=alertmanager/amop-test-routing namespace=alertmanager alertmanager=main-b

Anything else?

The current work-around is to duplicate referenced properties in each config, but that blows up the routing tree. We currently produce dozens of AM configs and will potentially get into hundreds, many of which will need to reference mute time intervals like "public holidays" that will take a lot of space and breach the 1MB k8s secret limit

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions