Mentioned in SAL (#wikimedia-operations) [2025-12-18T22:11:19Z] <cwhite@deploy2002> Finished deploy [statsv/statsv@0751b0b]: T383563 (duration: 00m 10s)

Mentioned in SAL (#wikimedia-operations) [2025-12-18T22:11:09Z] <cwhite@deploy2002> Started deploy [statsv/statsv@0751b0b]: T383563

Change #1217213 merged by jenkins-bot:

[performance/statsv@master] statsv: Add support for histograms

https://gerrit.wikimedia.org/r/1217213

@Krinkle @colewhite I've added you both as reviewers for the performance/statsv change needed for this. It has tests, but I don't actually know that domain very well, so any guidance would be appreciated if I still got something wrong.

Change #1217213 had a related patch set uploaded (by Michael Große; author: Michael Große):

[performance/statsv@master] statsv: Add support for histograms

https://gerrit.wikimedia.org/r/1217213

In T383563#11434840, @colewhite wrote:

In T383563#11434353, @Michael wrote:

However, I lack the knowledge about what happens afterward. dogstatsd seems to support metrics (|h), but I'm not sure what it does with that and how it is passed on to the next part of our pipeline. And we also need to somehow define the buckets to use.

The "dogstatsd native" histogram type passes control of bucket definitions to infrastructure configuration. This added friction and shared nature makes the option less appealing than at first glance.

In T411474#11435209, @bd808 wrote:

@colewhite is there a way to search for all of the dashboards that have graphite annotations? I fixed the 3 that @hashar explicitly listed, but I'm not excited enough about this to want to manually check every dashboard.

@colewhite is there a way to search for all of the dashboards that have graphite annotations? I fixed the 3 that @hashar explicitly listed, but I'm not excited enough about this to want to manually check every dashboard.

In T411474#11434904, @colewhite wrote:

For awareness, there's also a regression affecting annotations in Grafana: https://github.com/grafana/grafana/issues/110265

For awareness, there's also a regression affecting annotations in Grafana: https://github.com/grafana/grafana/issues/110265

In T383563#11434353, @Michael wrote:

However, I lack the knowledge about what happens afterward. dogstatsd seems to support metrics (|h), but I'm not sure what it does with that and how it is passed on to the next part of our pipeline. And we also need to somehow define the buckets to use.

I'm now encountering this issue again for our current OKR work on the Revise Tone Structured Task and tracking how well we can match paragraphs (T407031).

Change #1215241 had a related patch set uploaded (by Michael Große; author: Michael Große):

[mediawiki/extensions/WikimediaEvents@master] feat(mw.track): add support for histogram metrics

https://gerrit.wikimedia.org/r/1215241

I took a shot at fixing https://grafana-rw.wikimedia.org/d/35WSHOjVk/application-servers-red-k8s. I think it works. I used {channel="scap"} |~ "Finished scap sync-world|Synchronized" for the MW deploy query. Just "(?i)finished|synchronized" seemed to pick up helmfile deployments, or maybe they were legacy scap3 things?

@Michael pointed to https://grafana-rw.wikimedia.org/d/vGq7hbnMz/special3a-homepage-and-suggested-edits as a dashboard with some working annotations.

Note from @colewhite, the scap logs are available at (requires NDA login) https://grafana.wikimedia.org/goto/XO75YxZDR?orgId=1

Scap has:

def increment_stat(self, stat, all_stat=True, value=1):
    """Increment a stat in deploy.*

The replacement for this annotation tool is to use the Public Logs datasource in Grafana which is backed by Loki. Please let us know if the Observability team can be of further assistance.

Deployed and confirmed

Change #1202200 merged by jenkins-bot:

[operations/deployment-charts@master] api-gateway: Fix regex for api-gateway metrics

https://gerrit.wikimedia.org/r/1202200

Change #1202200 had a related patch set uploaded (by Clément Goubert; author: Clément Goubert):

[operations/deployment-charts@master] api-gateway: Fix regex for api-gateway metrics

https://gerrit.wikimedia.org/r/1202200

Change #1201599 merged by jenkins-bot:

[operations/deployment-charts@master] api-gateway: improve metrics mapping

https://gerrit.wikimedia.org/r/1201599

Change #1201599 had a related patch set uploaded (by Daniel Kinzler; author: Daniel Kinzler):

[operations/deployment-charts@master] api-gateway: improve metrics mapping

https://gerrit.wikimedia.org/r/1201599

This is what thanos shows for the ratelimit_service_rate_limit_wikimedia_route_name_default_rate_client_id prefix...

Hi folks, I'm working on updating the Grafana alerts Wikitech section. It's still a WIP but I'd greatly appreciate your feedback:

Change #1196533 merged by Andrea Denisse:

[operations/puppet@production] alertmanager: Add Slack route for the rweb team

https://gerrit.wikimedia.org/r/1196533

Great! So from the chat: we have #performance-alerts and then #test-platform-tools-reporting. The test-platform-tools-reporting is the channel where the Test Platform team send their alerts.

Change #1196533 had a related patch set uploaded (by Andrea Denisse; author: Andrea Denisse):

[operations/puppet@production] alertmanager: Add Slack route for the rweb team

https://gerrit.wikimedia.org/r/1196533

In T406689#11277602, @Peter wrote:

#api-alerts channel

Ah cool, didn't know, I'll try that out, thank you!

#api-alerts channel

Ah cool, didn't know, I'll try that out, thank you!