Component(s)

Prometheus

What is missing? Please describe.

Currently the recording rule assigning endpoint to shards is hardcoded to __address__.

We would like to be able to change the label and/or the method to, for example, allow zone aware sharding.
If all endpoints expose a certain label, let's say zone, either directly or through a rewrite rules, one could use this label instead of __address__. This would allow more control over how endpoints are distributed amongst the available shards.

In addition to this it would be nice to switch from hash based sharding to "value based sharding" where each shard is reacting to a specific value. This could be used to, for example, assign a specific shard to a specific zone, removing the need to find out which shard is scraping which zone after hashing.

Describe alternatives you've considered.

We're not aware of any alternative method to inject a "per-shard" rewrite rule to all metrics handled by a specific shard.

Environment Information.

Environment

Kubernetes Version: -
Prometheus-Operator Version: 0.72.0