Skip to content

[FEATURE] --output otel: cloud inventory as OpenTelemetry log records (per-row OTLP/JSON, snapshots over time, OTLP/HTTP push) #738

Closed

Description

@jeffreyaven

Summary

Add otel as a general --output format for stackql exec (and stackql shell), so a query's result set is emitted as OpenTelemetry log records in the OTLP/JSON encoding. The MCP server gained the same encoding for its audit log in #729 / #732; this issue makes it a first-class output format for the CLI and CI, then adds a push exporter so the records land directly in an OTLP-capable backend.

The intent is "your cloud config snapshot as telemetry": run an inventory query on a schedule, ship every row as a timestamped, structured record, and let the backend answer "what does the estate look like now", "what changed since the last run", and "who has had which entitlement, and since when". Target backends are systems that ingest OTLP logs and index arbitrary attributes: ClickHouse (ClickStack or the ClickHouse OTel exporter), Datadog, and the OTel Collector in front of anything else. The consumer is a machine, not a person, so complex payloads are expected and the grain should be as fine as possible.

This is distinct from the MCP audit log. The audit answers "what did the agent do" and never carries result rows. --output otel is the result set: rows are the point, and the operator asked for them.

What already exists

  • pkg/sink.NewOTelSink (mcp: adopt protocol revision 2026-07-28 and add otel audit log format; fix build date stamping (#729, #704) #732) is a generic decorator that wraps any Sink and encodes any payload as OTLP/JSON LogsData: envelope (resource, scope, schemaUrl), typed attributes, severity text, trace/span ids with per-key correlation, and a generic fallback that turns a JSON object's top-level fields into attributes and the object into the body. A payload can implement sink.OTelRecorder to supply its own records. Verified ingestible by otelcol-contrib 0.160.0 via the otlp_json_file receiver with no transform processor.
  • The output formats are dispatched in one switch in internal/stackql/output/output.go; the jsonl writer there streams rows one line at a time from the result stream and flushes per row. The otel writer is the same shape.
  • --output is a root persistent flag, so stackql shell gets the format for free.

Design

Grain

One log record per result row. A row is already the entity (a bucket, an instance, one IAM binding), and the projection defines the drift scope: the columns selected are the configuration being tracked. To track entitlements as entities, explode bindings to one row per principal and role in SQL; the telemetry follows the result grain.

Timestamps and snapshots

  • Every row of a statement carries the same timeUnixNano: the statement start time. That is the snapshot instant. Rows arrive across pages over several seconds, and as-of queries, diffs and latest-state queries break if each row has its own clock. observedTimeUnixNano records actual emission.
  • stackql.snapshot.id (uuid per statement execution) on every row.
  • A final completion record per statement: stackql.snapshot.complete=true, stackql.rows_returned, stackql.duration_ms. It tells the backend the snapshot is whole, and a zero-row snapshot still emits it, so "the resource is gone" is visible as drift rather than as silence.

Record shape

  • Body: the complete row as compact JSON, nested objects intact (iamConfiguration, tag maps, policy documents). Source of truth for the full document; parseable into a typed column or JSON type by a materialized view downstream.
  • Attributes: each scalar column as a typed attribute under its own column name (string / int / double / bool, sql.Null* wrappers unwrapped as the jsonl writer does); nested columns as JSON strings (portable across backends whose default attribute schema is string-valued). Reserved context under stackql.*:
    • stackql.snapshot.id
    • stackql.query (verbatim SQL, the collection recipe)
    • stackql.query.hash (series key for a recurring job, cheaper to group on than the SQL text)
    • stackql.provider, stackql.service, stackql.resource (from the statement's primary table; omitted for joins and views that span tables)
    • stackql.row.index
    • stackql.row.fingerprint: SHA-256 of the canonical row JSON, so drift is one comparison per identity instead of a comparison over a map of attributes
  • Resource attributes: service.name=stackql, service.version; the OTel cloud semantic conventions where derivable from WHERE parameters (cloud.provider, cloud.account.id from project / subscriptionId, cloud.region from region); and the standard OTEL_RESOURCE_ATTRIBUTES / OTEL_SERVICE_NAME environment variables so CI can stamp deployment.environment, job name and account tags with no new flags.
  • Correlation: one trace id per invocation, honouring TRACEPARENT from the environment (the CLI convention; trivially set in a pipeline step), otherwise generated; one span id per statement, shared by its rows, so "all rows of this statement" is a single filter.
  • Severity: INFO for rows and the completion record; ERROR with error.type / error.message for a failed statement, mirroring how WriteError produces a row today.
  • Scope: a new instrumentation scope for the CLI writer with its own pinned attribute schema version and a unit test asserting the exact attribute set, separate from the MCP audit's 1.0.0.

Identity

No heuristic resource identity in v1. The fingerprint is the value and the backend chooses the key (arn, id, selfLink, or a tuple of attributes). If a canonical stackql.resource.id proves necessary, a per-statement query hint in the style the engine already uses for /*+ AWAIT */ could declare identity columns; out of scope here.

Transport

  • Phase 1 (this issue): --output otel to stdout or -f outfile, one LogsData per row, streamed and flushed like jsonl. Consumable now by a collector's otlp_json_file receiver, which is enough for a CI step that ships a file.
  • Phase 2 (this issue, second PR): OTLP/HTTP push exporter driven by the standard OTEL_EXPORTER_OTLP_LOGS_ENDPOINT and OTEL_EXPORTER_OTLP_HEADERS variables, batching per statement (chunked for large results). The LogsData emitted is byte for byte the body of POST /v1/logs, so this is an HTTP client plus batching, no new dependency. Targets ClickStack's OTLP intake and Datadog's OTLP endpoint directly.
  • Phase 3 (docs): backend recipes for the joint use cases: a ClickHouse materialized view from otel_logs into a typed inventory table keyed by (snapshot_id, identity), then latest state per resource (argMax by timestamp), drift between two snapshots (join on identity, fingerprint inequality, anti-joins for appeared and disappeared), and entitlement history (principal, role, first seen, last seen).

Example

TRACEPARENT=00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01 \
OTEL_RESOURCE_ATTRIBUTES=deployment.environment=prod,stackql.job=s3-inventory \
stackql exec --output otel -f inventory.otlp.jsonl \
  "select name, region, encryption, public_access_blocked from aws.s3.buckets where region = 'us-east-1'"

One row (pretty-printed; the file holds it on one line):

{"resourceLogs":[{"resource":{"attributes":[
  {"key":"service.name","value":{"stringValue":"stackql"}},
  {"key":"service.version","value":{"stringValue":"0.10.624"}},
  {"key":"cloud.provider","value":{"stringValue":"aws"}},
  {"key":"cloud.region","value":{"stringValue":"us-east-1"}},
  {"key":"deployment.environment","value":{"stringValue":"prod"}},
  {"key":"stackql.job","value":{"stringValue":"s3-inventory"}}
]},"scopeLogs":[{"scope":{"name":"github.com/stackql/stackql/internal/stackql/output","version":"1.0.0"},
"logRecords":[{
  "timeUnixNano":"1788739200000000000","observedTimeUnixNano":"1788739203412000000",
  "severityNumber":9,"severityText":"INFO",
  "body":{"stringValue":"{\"encryption\":{\"algorithm\":\"AES256\"},\"name\":\"logs-bucket\",\"public_access_blocked\":true,\"region\":\"us-east-1\"}"},
  "attributes":[
    {"key":"name","value":{"stringValue":"logs-bucket"}},
    {"key":"region","value":{"stringValue":"us-east-1"}},
    {"key":"encryption","value":{"stringValue":"{\"algorithm\":\"AES256\"}"}},
    {"key":"public_access_blocked","value":{"boolValue":true}},
    {"key":"stackql.snapshot.id","value":{"stringValue":"1e0d4a6c-4b0f-4b4e-9f7d-2a6c9a1f3b21"}},
    {"key":"stackql.query","value":{"stringValue":"select name, region, encryption, public_access_blocked from aws.s3.buckets where region = 'us-east-1'"}},
    {"key":"stackql.query.hash","value":{"stringValue":"9c1d2e..."}},
    {"key":"stackql.provider","value":{"stringValue":"aws"}},
    {"key":"stackql.service","value":{"stringValue":"s3"}},
    {"key":"stackql.resource","value":{"stringValue":"buckets"}},
    {"key":"stackql.row.index","value":{"intValue":"0"}},
    {"key":"stackql.row.fingerprint","value":{"stringValue":"sha256:7f3a..."}}
  ],
  "traceId":"4bf92f3577b34da6a3ce929d0e0e4736","spanId":"a1b2c3d4e5f60718"
}],"schemaUrl":"https://opentelemetry.io/schemas/1.44.0"}],"schemaUrl":"https://opentelemetry.io/schemas/1.44.0"}]}

Followed by the completion record for the statement (same snapshot id, trace and span; body snapshot complete aws.s3.buckets; stackql.snapshot.complete=true, stackql.rows_returned, stackql.duration_ms).

Tasks

  • otel case in output.GetOutputWriter; writer streams one LogsData per row through sink.NewOTelSink over an io.Writer-backed sink; completion record per statement; error record on failure
  • Snapshot timestamp (statement start) on every row; stackql.snapshot.id; span id per statement; trace id per invocation with TRACEPARENT honoured
  • Attribute mapping: scalar columns typed, nested columns as JSON strings, stackql.* context, stackql.row.fingerprint; body = full row JSON
  • Resource attributes: service.*, derived cloud.*, OTEL_RESOURCE_ATTRIBUTES / OTEL_SERVICE_NAME
  • Pinned scope version with a schema test; --output flag help and global-flags docs updated (json | jsonl | table | csv | text | otel)
  • Robot scenario: stackql exec --output otel against a mocked provider, asserting per-row records, shared snapshot timestamp and id, fingerprint, completion record, and the zero-row completion record
  • Verify the file against a stock collector (otlp_json_file receiver) as done for the MCP audit
  • Phase 2: OTLP/HTTP push exporter (OTEL_EXPORTER_OTLP_LOGS_ENDPOINT, OTEL_EXPORTER_OTLP_HEADERS), batched per statement, with a mocked-endpoint robot scenario (tracked in [FEATURE] --output otel phase 2 and 3: OTLP/HTTP push exporter and backend recipes #755)
  • Phase 3: docs page with the ClickHouse materialized view, latest-state, drift and entitlement-history queries, and the Datadog resource mapping (tracked in [FEATURE] --output otel phase 2 and 3: OTLP/HTTP push exporter and backend recipes #755)

Acceptance criteria

  • Every row of a statement shares timeUnixNano, stackql.snapshot.id and spanId; a second statement in the same invocation gets a new snapshot id and span under the same trace
  • A zero-row statement emits exactly one record: the completion record
  • Re-running an identical query against unchanged mock data yields identical stackql.row.fingerprint values; changing one column value changes only that row's fingerprint
  • Output is accepted by a stock OTel Collector otlp_json_file receiver with no transform processor
  • jsonl, json, csv, table and text output are byte-identical to today
  • Attribute schema is pinned by a unit test and documented

Decisions taken

  • Nested columns are JSON strings in attributes, with the full document in the body, rather than OTLP kvlistValue structures: more portable across backends whose default attribute schema is string-valued. Revisit if a structured-attribute schema becomes the norm.
  • Completion is a separate record rather than a field on the last row, so the empty-snapshot case is covered.
  • No identity heuristics; fingerprint plus backend-chosen key.

Out of scope

Activity

  1. self-assigned this
    on Sep 6, 2026
  2. jeffreyaven commented on Sep 7, 2026

    @jeffreyaven
    MemberAuthor

    Phase 1 shipped in #754 (main 828e1f0, going out in the v0.11 release): otel is a general --output format for exec and shell, one OTLP/JSON LogsData per row plus a completion record per statement, with the snapshot, attribute and resource semantics described above. Documented under Output Modes on stackql.io.

    All phase 1 tasks and every acceptance criterion are met, including the collector check: otelcol-contrib 0.160.0 with the otlp_json_file receiver and no transform processor ingests the file with zero errors, typed attributes intact, TRACEPARENT honoured, and both completion records present (8 rows and the zero-row statement).

    Phase 2 (OTLP/HTTP push exporter) and phase 3 (ClickHouse / Datadog recipes) are tracked in #755.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

enhancementNew feature or request

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions