In SDP 24.7 we upgraded to JMX Exporter 1.0.1. Unfortunately this caused the performance of the metrics endpoints to degrade severly. We tracked the issue down to a piece of code in the Prometheus Java Client which has since been fixed:

• Simple MetricsSnapshot performance change prometheus/client_java#963
• additional improve MetricSnapshots.Builder performance prometheus/client_java#985

For our next release we need to fix this and we see these options:

Option 1: New upstream JMX Exporter

This requires a new client_java release first which we hope to see in the week of October 14 according to a thread on Slack. When that is done we should put up a PR with jmx_exporter upgrading the client java version and then nicely ask if anyone is up for a new jmx_exporter release. In a thread on Slack one of the maintainers said that they'd like to get OpenTelemetry support in for the next release. If that is the case a release might be a bit off and I hope that we can ask nicely for a bugfix release 1.0.2 instead.

Option 2: Revert to JMX Exporter 0.20

This is an option we have and it'd be relatively easy to do (but requires some changes as the metrics path changed) but we'd like to avoid downgrading a dependency in case vulnerabilities are discovered.

Option 3: Build a patched JMX Exporter ourselves

We tried building a JMX Exporter with the current main branch of client_java and that fixes the performance issues so we know that the fixes are good. We could build the exporter from source if needed.