Skip to content

Pick one OTel-based agent instead of per-framework skip checks #1469

Open

Description

@stokpop

Problem

The OTel-based frameworks (OpenTelemetry, Splunk OTel, Elastic OTel) detect themselves independently, so several of them can be installed into the same JVM. This risks duplicate telemetry and conflicting instrumentation. Examples:

Setup Agents installed
OTEL_EXPORTER_OTLP_ENDPOINT set during staging + otel-collector binding Splunk + OpenTelemetry (already on main)
Elastic OTel binding + OTEL_EXPORTER_OTLP_ENDPOINT set during staging Splunk + Elastic OTel
Elastic OTel binding without credentials OpenTelemetry instead of Elastic OTel (name matches otel)
ELASTIC_OTEL_AGENT + otel-collector binding OpenTelemetry + Elastic OTel

Conflicts are currently prevented by skip checks spread across frameworks: OpenTelemetry checks for an Elastic service, and Splunk checks ELASTIC_OTEL_AGENT. Each new OTel distribution adds more of them (see #1360).

Proposal

The OTel-based frameworks share one exclusive group, and each reports how specific its detection was. The registry keeps one of them, in this priority order:

  1. Explicit env selection (ELASTIC_OTEL_AGENT, SPLUNK_OTEL_AGENT)
  2. Specific service binding (elastic-otel, splunk, ...)
  3. Generic signals (an otel name or tag, OTEL_EXPORTER_OTLP_* env vars)

On a tie, the registry logs a warning naming the skipped agents. The existing per-framework skip checks can then be removed.

Non-OTel agents (for example Datadog) are out of scope and keep detecting independently, since running them next to an OTel agent can be intentional.

Related: env vs binding precedence

Elastic OTel lets OTEL_EXPORTER_OTLP_ENDPOINT and OTEL_EXPORTER_OTLP_HEADERS set during staging override the binding values, while OTEL_SERVICE_NAME does not. The order should be the same for all keys and all OTel-based frameworks.

Decisions

  • Allow more than one OTel agent? Proposal: only when more than one is selected explicitly via env (for example both ELASTIC_OTEL_AGENT and SPLUNK_OTEL_AGENT): install all of them and log a warning. Detection by binding or generic signals never installs more than one. Rationale: OpenTelemetry supports a single Java agent per JVM (duplicate spans, competing global instance). The recommended way to send data to multiple backends is a single agent exporting to an OTel Collector that fans out.

Refs #1360

Activity

  1. stokpop commented on Nov 13, 2026

    @stokpop
    ContributorAuthor

    Before I start on this, I would like to settle three points. I read the current detection code to check them:

    1. More than one OTel agent (the open checkbox above). I agree with the proposal: more than one only when several are selected explicitly by env (ELASTIC_OTEL_AGENT and SPLUNK_OTEL_AGENT), install all and log a warning. Binding or generic signals never install more than one.
    2. Elastic OTel binding with missing credentials. Today ElasticOtelJavaAgentFramework.Detect() returns nothing when the endpoint or credentials are missing, so a generic OpenTelemetry (name matches otel) takes over silently. I suggest that Elastic still counts as the specific match and logs which keys are missing, so the generic agent does not win by accident. This is the third row of the table.
    3. One precedence order for env vs binding. Today it differs: Elastic lets OTEL_EXPORTER_OTLP_ENDPOINT and OTEL_EXPORTER_OTLP_HEADERS from the environment override the binding, but the binding wins for otel.service.name; Splunk reads the environment first; OpenTelemetry reads the binding only. I suggest that an explicit environment value wins over the binding for all keys and all three frameworks. That is a small behaviour change for the service name in Elastic.

    If these are fine, I plan two PRs:

    • PR 1: the exclusive group in the registry (one match per group by priority, a warning on a tie), the old skip checks removed, and unit tests for the four combinations from the table.
    • PR 2: the single env-over-binding precedence, with unit tests per key.

    Integration tests for the combinations (checking the running JVM's -javaagent and -Dotel.* values) would follow once the test improvements I am working on are in.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions