Repository navigation
Description
Summary
Add otel as a general --output format for stackql exec (and stackql shell), so a query's result set is emitted as OpenTelemetry log records in the OTLP/JSON encoding. The MCP server gained the same encoding for its audit log in #729 / #732; this issue makes it a first-class output format for the CLI and CI, then adds a push exporter so the records land directly in an OTLP-capable backend.
The intent is "your cloud config snapshot as telemetry": run an inventory query on a schedule, ship every row as a timestamped, structured record, and let the backend answer "what does the estate look like now", "what changed since the last run", and "who has had which entitlement, and since when". Target backends are systems that ingest OTLP logs and index arbitrary attributes: ClickHouse (ClickStack or the ClickHouse OTel exporter), Datadog, and the OTel Collector in front of anything else. The consumer is a machine, not a person, so complex payloads are expected and the grain should be as fine as possible.
This is distinct from the MCP audit log. The audit answers "what did the agent do" and never carries result rows. --output otel is the result set: rows are the point, and the operator asked for them.
What already exists
pkg/sink.NewOTelSink(mcp: adopt protocol revision 2026-07-28 and add otel audit log format; fix build date stamping (#729, #704) #732) is a generic decorator that wraps anySinkand encodes any payload as OTLP/JSONLogsData: envelope (resource, scope,schemaUrl), typed attributes, severity text, trace/span ids with per-key correlation, and a generic fallback that turns a JSON object's top-level fields into attributes and the object into the body. A payload can implementsink.OTelRecorderto supply its own records. Verified ingestible by otelcol-contrib 0.160.0 via theotlp_json_filereceiver with no transform processor.- The output formats are dispatched in one
switchininternal/stackql/output/output.go; thejsonlwriter there streams rows one line at a time from the result stream and flushes per row. Theotelwriter is the same shape. --outputis a root persistent flag, sostackql shellgets the format for free.
Design
Grain
One log record per result row. A row is already the entity (a bucket, an instance, one IAM binding), and the projection defines the drift scope: the columns selected are the configuration being tracked. To track entitlements as entities, explode bindings to one row per principal and role in SQL; the telemetry follows the result grain.
Timestamps and snapshots
- Every row of a statement carries the same
timeUnixNano: the statement start time. That is the snapshot instant. Rows arrive across pages over several seconds, and as-of queries, diffs and latest-state queries break if each row has its own clock.observedTimeUnixNanorecords actual emission. stackql.snapshot.id(uuid per statement execution) on every row.- A final completion record per statement:
stackql.snapshot.complete=true,stackql.rows_returned,stackql.duration_ms. It tells the backend the snapshot is whole, and a zero-row snapshot still emits it, so "the resource is gone" is visible as drift rather than as silence.
Record shape
- Body: the complete row as compact JSON, nested objects intact (
iamConfiguration, tag maps, policy documents). Source of truth for the full document; parseable into a typed column or JSON type by a materialized view downstream. - Attributes: each scalar column as a typed attribute under its own column name (string / int / double / bool,
sql.Null*wrappers unwrapped as thejsonlwriter does); nested columns as JSON strings (portable across backends whose default attribute schema is string-valued). Reserved context understackql.*:stackql.snapshot.idstackql.query(verbatim SQL, the collection recipe)stackql.query.hash(series key for a recurring job, cheaper to group on than the SQL text)stackql.provider,stackql.service,stackql.resource(from the statement's primary table; omitted for joins and views that span tables)stackql.row.indexstackql.row.fingerprint: SHA-256 of the canonical row JSON, so drift is one comparison per identity instead of a comparison over a map of attributes
- Resource attributes:
service.name=stackql,service.version; the OTel cloud semantic conventions where derivable fromWHEREparameters (cloud.provider,cloud.account.idfromproject/subscriptionId,cloud.regionfromregion); and the standardOTEL_RESOURCE_ATTRIBUTES/OTEL_SERVICE_NAMEenvironment variables so CI can stampdeployment.environment, job name and account tags with no new flags. - Correlation: one trace id per invocation, honouring
TRACEPARENTfrom the environment (the CLI convention; trivially set in a pipeline step), otherwise generated; one span id per statement, shared by its rows, so "all rows of this statement" is a single filter. - Severity:
INFOfor rows and the completion record;ERRORwitherror.type/error.messagefor a failed statement, mirroring howWriteErrorproduces a row today. - Scope: a new instrumentation scope for the CLI writer with its own pinned attribute schema version and a unit test asserting the exact attribute set, separate from the MCP audit's
1.0.0.
Identity
No heuristic resource identity in v1. The fingerprint is the value and the backend chooses the key (arn, id, selfLink, or a tuple of attributes). If a canonical stackql.resource.id proves necessary, a per-statement query hint in the style the engine already uses for /*+ AWAIT */ could declare identity columns; out of scope here.
Transport
- Phase 1 (this issue):
--output otelto stdout or-f outfile, oneLogsDataper row, streamed and flushed likejsonl. Consumable now by a collector'sotlp_json_filereceiver, which is enough for a CI step that ships a file. - Phase 2 (this issue, second PR): OTLP/HTTP push exporter driven by the standard
OTEL_EXPORTER_OTLP_LOGS_ENDPOINTandOTEL_EXPORTER_OTLP_HEADERSvariables, batching per statement (chunked for large results). TheLogsDataemitted is byte for byte the body ofPOST /v1/logs, so this is an HTTP client plus batching, no new dependency. Targets ClickStack's OTLP intake and Datadog's OTLP endpoint directly. - Phase 3 (docs): backend recipes for the joint use cases: a ClickHouse materialized view from
otel_logsinto a typed inventory table keyed by(snapshot_id, identity), then latest state per resource (argMaxby timestamp), drift between two snapshots (join on identity, fingerprint inequality, anti-joins for appeared and disappeared), and entitlement history (principal, role, first seen, last seen).
Example
TRACEPARENT=00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01 \
OTEL_RESOURCE_ATTRIBUTES=deployment.environment=prod,stackql.job=s3-inventory \
stackql exec --output otel -f inventory.otlp.jsonl \
"select name, region, encryption, public_access_blocked from aws.s3.buckets where region = 'us-east-1'"One row (pretty-printed; the file holds it on one line):
{"resourceLogs":[{"resource":{"attributes":[
{"key":"service.name","value":{"stringValue":"stackql"}},
{"key":"service.version","value":{"stringValue":"0.10.624"}},
{"key":"cloud.provider","value":{"stringValue":"aws"}},
{"key":"cloud.region","value":{"stringValue":"us-east-1"}},
{"key":"deployment.environment","value":{"stringValue":"prod"}},
{"key":"stackql.job","value":{"stringValue":"s3-inventory"}}
]},"scopeLogs":[{"scope":{"name":"github.com/stackql/stackql/internal/stackql/output","version":"1.0.0"},
"logRecords":[{
"timeUnixNano":"1788739200000000000","observedTimeUnixNano":"1788739203412000000",
"severityNumber":9,"severityText":"INFO",
"body":{"stringValue":"{\"encryption\":{\"algorithm\":\"AES256\"},\"name\":\"logs-bucket\",\"public_access_blocked\":true,\"region\":\"us-east-1\"}"},
"attributes":[
{"key":"name","value":{"stringValue":"logs-bucket"}},
{"key":"region","value":{"stringValue":"us-east-1"}},
{"key":"encryption","value":{"stringValue":"{\"algorithm\":\"AES256\"}"}},
{"key":"public_access_blocked","value":{"boolValue":true}},
{"key":"stackql.snapshot.id","value":{"stringValue":"1e0d4a6c-4b0f-4b4e-9f7d-2a6c9a1f3b21"}},
{"key":"stackql.query","value":{"stringValue":"select name, region, encryption, public_access_blocked from aws.s3.buckets where region = 'us-east-1'"}},
{"key":"stackql.query.hash","value":{"stringValue":"9c1d2e..."}},
{"key":"stackql.provider","value":{"stringValue":"aws"}},
{"key":"stackql.service","value":{"stringValue":"s3"}},
{"key":"stackql.resource","value":{"stringValue":"buckets"}},
{"key":"stackql.row.index","value":{"intValue":"0"}},
{"key":"stackql.row.fingerprint","value":{"stringValue":"sha256:7f3a..."}}
],
"traceId":"4bf92f3577b34da6a3ce929d0e0e4736","spanId":"a1b2c3d4e5f60718"
}],"schemaUrl":"https://opentelemetry.io/schemas/1.44.0"}],"schemaUrl":"https://opentelemetry.io/schemas/1.44.0"}]}Followed by the completion record for the statement (same snapshot id, trace and span; body snapshot complete aws.s3.buckets; stackql.snapshot.complete=true, stackql.rows_returned, stackql.duration_ms).
Tasks
-
otelcase inoutput.GetOutputWriter; writer streams oneLogsDataper row throughsink.NewOTelSinkover anio.Writer-backed sink; completion record per statement; error record on failure - Snapshot timestamp (statement start) on every row;
stackql.snapshot.id; span id per statement; trace id per invocation withTRACEPARENThonoured - Attribute mapping: scalar columns typed, nested columns as JSON strings,
stackql.*context,stackql.row.fingerprint; body = full row JSON - Resource attributes:
service.*, derivedcloud.*,OTEL_RESOURCE_ATTRIBUTES/OTEL_SERVICE_NAME - Pinned scope version with a schema test;
--outputflag help andglobal-flagsdocs updated (json | jsonl | table | csv | text | otel) - Robot scenario:
stackql exec --output otelagainst a mocked provider, asserting per-row records, shared snapshot timestamp and id, fingerprint, completion record, and the zero-row completion record - Verify the file against a stock collector (
otlp_json_filereceiver) as done for the MCP audit - Phase 2: OTLP/HTTP push exporter (
OTEL_EXPORTER_OTLP_LOGS_ENDPOINT,OTEL_EXPORTER_OTLP_HEADERS), batched per statement, with a mocked-endpoint robot scenario (tracked in [FEATURE]--output otelphase 2 and 3: OTLP/HTTP push exporter and backend recipes #755) - Phase 3: docs page with the ClickHouse materialized view, latest-state, drift and entitlement-history queries, and the Datadog resource mapping (tracked in [FEATURE]
--output otelphase 2 and 3: OTLP/HTTP push exporter and backend recipes #755)
Acceptance criteria
- Every row of a statement shares
timeUnixNano,stackql.snapshot.idandspanId; a second statement in the same invocation gets a new snapshot id and span under the same trace - A zero-row statement emits exactly one record: the completion record
- Re-running an identical query against unchanged mock data yields identical
stackql.row.fingerprintvalues; changing one column value changes only that row's fingerprint - Output is accepted by a stock OTel Collector
otlp_json_filereceiver with no transform processor -
jsonl,json,csv,tableandtextoutput are byte-identical to today - Attribute schema is pinned by a unit test and documented
Decisions taken
- Nested columns are JSON strings in attributes, with the full document in the body, rather than OTLP
kvlistValuestructures: more portable across backends whose default attribute schema is string-valued. Revisit if a structured-attribute schema becomes the norm. - Completion is a separate record rather than a field on the last row, so the empty-snapshot case is covered.
- No identity heuristics; fingerprint plus backend-chosen key.
Out of scope
- Metrics and spans (log signal only)
- Backend-specific sinks or schemas inside stackql (recipes live in docs)
- The MCP audit log format, which stays as shipped in mcp: adopt protocol revision 2026-07-28 and add otel audit log format; fix build date stamping (#729, #704) #732 and keeps its no-rows redaction rule
Activity
v0.11 release prep: --output otel inventory snapshots (#738), SHOW CO…
jeffreyaven commentedon Sep 7, 2026
Phase 1 shipped in #754 (main
828e1f0, going out in the v0.11 release):otelis a general--outputformat forexecandshell, one OTLP/JSONLogsDataper row plus a completion record per statement, with the snapshot, attribute and resource semantics described above. Documented under Output Modes on stackql.io.All phase 1 tasks and every acceptance criterion are met, including the collector check: otelcol-contrib 0.160.0 with the
otlp_json_filereceiver and no transform processor ingests the file with zero errors, typed attributes intact,TRACEPARENThonoured, and both completion records present (8 rows and the zero-row statement).Phase 2 (OTLP/HTTP push exporter) and phase 3 (ClickHouse / Datadog recipes) are tracked in #755.