Repository navigation
feat(aws-lambda): Add queue.process spans for messaging triggers - #8223
Conversation
A synchronous boto3 Invoke now carries sentry-trace and baggage in ClientContext.custom (added with the Lambda service extension). The Lambda integration continues the trace from there. ClientContext wins over event headers, because a forwarded HTTP event can keep headers from further upstream. The ClientContextTrace test function works around sam local, which drops ClientContext, by handing the event's client_context to the runtime. Co-Authored-By: Johannes Daxböck <114897+dingsdax@users.noreply.github.com>
Classify the delivered event into faas.trigger: http (API Gateway, function URLs, ALB, CloudFront, AppSync), pubsub (SQS, SNS, Kinesis, Kafka, MQ, SES, EventBridge), datasource (S3, DynamoDB Streams, DocumentDB), timer (EventBridge schedules) or other. The trigger is read from the event envelope, never from its payload. messaging.batch.message_count is now only set for messaging deliveries, with the number of delivered messages. Co-Authored-By: Johannes Daxböck <114897+dingsdax@users.noreply.github.com>
For an SQS, SNS, Kafka, ActiveMQ or RabbitMQ delivery, the handler runs inside a queue.process child span (kind consumer) with messaging.system, messaging.operation.type and messaging.operation.name set to process, the batch message count, and the destination name when all records share it. The span links to each distinct producer context found in the records' Sentry message attributes: SQS messageAttributes, SNS MessageAttributes, and SQS bodies that wrap an SNS notification. Co-Authored-By: Johannes Daxböck <114897+dingsdax@users.noreply.github.com>
Coverage Results 📊✅ 58870 passed | ⏭️ 3071 skipped | Total: 61941 | Pass Rate: 95.04% | Execution Time: 158m 56s 📊 Comparison with Base Branch
➖ Removed Tests (1)View removed tests
All tests are passing successfully. ❌ Patch coverage is 12.50% (2 of 16 changed executable lines covered; target 80%). Changed files with executable lines (1)
Coverage diff@@ Coverage Diff @@
## master #8223 +/-##
==========================================
- Coverage 90.21% 90.12% -0.09%
==========================================
Files 203 200 -3
Tracked lines 27335 21872 -5463
Branches 10246 7628 -2618
==========================================
- Hits 24660 19711 -4949
+ Misses 2675 2161 -514
+ Partials 1559 1255 -304Generated by Coverage Action |
…a span Keep the count on messaging spans only (boto3 receive and batch-send spans, and queue.process for SQS/SNS triggers), as in the OTel messaging conventions and the JS SDK. This drops the batch counting for Kafka, MQ and RabbitMQ deliveries, and matches #8219 so both PRs change these lines the same way. Co-Authored-By: johdax <114897+dingsdax@users.noreply.github.com>
…-process Co-Authored-By: johdax <114897+dingsdax@users.noreply.github.com> # Conflicts: # sentry_sdk/integrations/aws_lambda/_handler.py # sentry_sdk/integrations/aws_lambda/_trigger.py
The error is captured, both the process span and the invocation span are marked as errors, and the exception still reaches the Lambda runtime. Co-Authored-By: johdax <114897+dingsdax@users.noreply.github.com>
Read client_context and its custom dict with getattr, so a context object without them falls back to the event headers. Co-Authored-By: Johannes Daxböck <114897+dingsdax@users.noreply.github.com>
…r/aws-lambda-trigger Co-Authored-By: Johannes Daxböck <114897+dingsdax@users.noreply.github.com>
Each case is a SAM invoke. The removed cases only checked other keys of the same lookup table. Co-Authored-By: Johannes Daxböck <114897+dingsdax@users.noreply.github.com>
…-process Co-Authored-By: Johannes Daxböck <114897+dingsdax@users.noreply.github.com>
OTel sets faas.trigger on the FaaS span, which already has it. Also make the producer link cap private, like the other module constants. Co-Authored-By: Johannes Daxböck <114897+dingsdax@users.noreply.github.com>
…a-invoke-trace-continuation # Conflicts: # sentry_sdk/integrations/boto3/_client.py # sentry_sdk/integrations/boto3/_instrumentation.py # sentry_sdk/integrations/boto3/_services/base.py # sentry_sdk/integrations/boto3/_services/lambda_.py # sentry_sdk/integrations/boto3/_services/registry.py
) ### Description Since #8137, #8142, and #8145, boto3 puts `sentry-trace` in SQS and SNS message attributes, but nothing reads it on the Lambda side. This PR is the smallest change that closes the loop for SQS- and SNS-triggered functions. The Lambda integration now reads `sentry-trace` from each record and adds a `message_origin` link from the invocation span to each distinct producer span. This matches what the boto3 `ReceiveMessage` span already does. Messages never become the invocation's parent. It reads: - SQS records: `messageAttributes["sentry-trace"].stringValue` - SNS records: `Sns.MessageAttributes["sentry-trace"].Value` - SNS → SQS without raw message delivery: the SNS envelope in the SQS `body`, the same as boto3 does Links are deduplicated, because a batch send gives many messages the same producer span, and capped at 128, OTel's default, because Lambda can deliver up to 10,000 SQS messages at once. **Scope.** This is intentionally kept separate from the stretch stack. It doesn't add `faas.trigger` (#8222) or `queue.process` spans (#8223), and it doesn't touch `messaging.batch.message_count` (#8229). The invocation span is otherwise unchanged. Synchronous `Invoke` propagation through `ClientContext` is #8221. That PR changes the `continue_trace` call and this one only adds `links=`, so the two are independent. The link-building code repeats boto3's `_message_link` in a Lambda-event shape: Lambda records use `messageAttributes` / `stringValue` / `body`, while the boto3 response uses `MessageAttributes` / `StringValue` / `Body`. Importing from the boto3 integration would make `aws_lambda` depend on botocore. **Testing.** `test_producer_links` sends SQS, SNS, and SNS-to-SQS events to `BasicOkSpanStreaming`. It asserts the segment span's `links` and that the segment span has no parent. It passes in CI's Cloud (Lambda) jobs, and `ruff` and `mypy sentry_sdk` pass. `codecov/patch` is red because the Lambda tests run in `sam local` containers, so their coverage isn't counted. #### Issues * Part of #8081 <!-- junior-request-attribution:start --> via **webb**, @sentrivana. <!-- junior-request-attribution:end --> <!-- junior-session-footer:start --> <!-- junior-conversation-id:slack%3AC09M26BC6A1%3A1794211222.463239 --> -- [View Session](https://junior-prod.sentry.dev/conversations/slack%3AC09M26BC6A1%3A1794211222.463239) [[Sentry]](https://sentry.sentry.io/explore/conversations/slack%3AC09M26BC6A1%3A1794211222.463239/?project=4510944073809921) <!-- junior-session-footer:end --> --------- Co-authored-by: sentry-junior[bot] <264270552+sentry-junior[bot]@users.noreply.github.com> Co-authored-by: Alexander Webb <alexander.webb@sentry.io> Co-authored-by: Ivana Kellyer <131587164+sentrivana@users.noreply.github.com>
Restores the batch count on the invocation span, as on major/3.0, so this PR only adds faas.trigger.
… span (#8229) ### Description The Lambda span set `messaging.batch.message_count` to `1` for every event that isn't a list, and to `len(event)` for a top-level list. An SQS or SNS delivery is a dict with `Records`, so a batch of 10 messages reported 1. HTTP calls and direct invocations reported a batch they don't have. This PR removes the attribute from the Lambda span. The count belongs on messaging spans, as in the OTel messaging conventions: - boto3 `ReceiveMessage` and batch-send spans already set it (#8137, #8142). - For SQS/SNS-triggered Lambdas, the `queue.process` span in #8223 sets it. The JS SDK doesn't set it on the Lambda span either. The attribute was added in 3.0 (#7941) and never shipped in 2.x, so no released behavior changes. Replaces #8219 (same commits). GitHub never ran CI on #8219. **Note for review:** this is narrower than PY-3015, which asks to keep the attribute for SQS/SNS deliveries. @alexander-alderman-webb, please confirm that's OK. #### Issues * resolves: PY-3015 <!-- junior-request-attribution:start --> via @dingsdax. <!-- junior-request-attribution:end --> <!-- junior-session-footer:start --> <!-- junior-conversation-id:slack%3AC0C9X9V5NMT%3A1793904671.980379 --> -- [View Session](https://junior-prod.sentry.dev/conversations/slack%3AC0C9X9V5NMT%3A1793904671.980379) [[Sentry]](https://sentry.sentry.io/explore/conversations/slack%3AC0C9X9V5NMT%3A1793904671.980379/?project=4510944073809921) <!-- junior-session-footer:end --> --------- Co-authored-by: sentry-junior[bot] <264270552+sentry-junior[bot]@users.noreply.github.com> Co-authored-by: Johannes Daxböck <114897+dingsdax@users.noreply.github.com>
…a-trigger # Conflicts: # tests/integrations/aws_lambda/test_aws_lambda.py
…-process # Conflicts: # sentry_sdk/integrations/aws_lambda/_trigger.py
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 8aefb75. Configure here.
…-process # Conflicts: # tests/integrations/aws_lambda/test_aws_lambda.py
### Description Sets `faas.trigger` on the Lambda span based on the shape of the delivered event: - `http`: API Gateway REST/HTTP, function URLs, ALB, CloudFront, AppSync - `pubsub`: SQS, SNS, Kinesis, Kafka, MQ, SES, EventBridge - `datasource`: S3, DynamoDB Streams, DocumentDB - `timer`: EventBridge schedules - `other`: anything else The trigger is read from the event envelope, never from its payload. `test_trigger` has one case per code path. Other event sources use the same lookup table, so they have no extra cases. This PR doesn't touch `messaging.batch.message_count`. That change is in #8229. This is Pablo's code from `pablo/aws-lambda-trigger`, without his batch counting for the invocation span. It's the second of three stacked PRs (#8221 → this → #8223). #### Issues * resolves: PY-3018 (`faas.trigger` part; the per-source attributes in the issue aren't in this PR) * Part of #8081 <!-- junior-request-attribution:start --> via @dingsdax. <!-- junior-request-attribution:end --> <!-- junior-session-footer:start --> <!-- junior-conversation-id:slack%3AC0C9X9V5NMT%3A1793904671.980379 --> -- [View Session](https://junior-prod.sentry.dev/conversations/slack%3AC0C9X9V5NMT%3A1793904671.980379) [[Sentry]](https://sentry.sentry.io/explore/conversations/slack%3AC0C9X9V5NMT%3A1793904671.980379/?project=4510944073809921) <!-- junior-session-footer:end --> --------- Co-authored-by: Pablo Deputter <pablo.deputter@sentry.io> Co-authored-by: Pablo Deputter <71842639+pabloDeputter@users.noreply.github.com> Co-authored-by: sentry-junior[bot] <264270552+sentry-junior[bot]@users.noreply.github.com> Co-authored-by: Johannes Daxböck <114897+dingsdax@users.noreply.github.com>
|
@sentry-junior Update this branch with latest major/3.0 and solve any merge conflicts |
…a-queue-process # Conflicts: # sentry_sdk/integrations/aws_lambda/_handler.py # sentry_sdk/integrations/aws_lambda/_trigger.py # tests/integrations/aws_lambda/test_aws_lambda.py
|
Done. I merged The conflicts were in |
#8257 links SQS/SNS invocations to their producers on the invocation span. Remove this PR's second copy of that code from _trigger.py, keep the queue.process span without links, and reuse the test helpers and records already on major/3.0. Co-Authored-By: Ivana Kellyer <131587164+sentrivana@users.noreply.github.com>
Following the OTel messaging conventions, the process span carries the links to each message's creation context. Move the links that #8257 put on the invocation span to the queue.process span, reusing the same _get_producer_links. Co-Authored-By: Ivana Kellyer <131587164+sentrivana@users.noreply.github.com>
|
Does this need a migration guide update? |
0cb05d4 to
1a11d5a
Compare
|
I don't think it needs one. Everything this PR adds is new, and nothing that 2.x users rely on is renamed or removed:
The one visible difference is that spans a user starts inside an SQS- or SNS-triggered handler now have |

Description
For an SQS or SNS delivery, the handler now runs inside a
queue.processchild span of kindconsumer. The span has:messaging.systemmessaging.operation.typeandmessaging.operation.name, bothprocessThis fills in the processing side of the Queues insight.
Producer links move to this span. The OTel messaging conventions put a consumer's links to each message's creation context on its process span. #8257 (merged) put them on the invocation span. This PR moves them: the invocation span has no links anymore, and
queue.processgets them. The links are built by the same_get_producer_linksfrom #8257 (dedupe, the cap of 128, skipping the continued trace), with no second copy of that parsing. The tests reuse theSQS_RECORD/SNS_RECORDfixtures and thetest_producer_linkscases already onmajor/3.0. Those tests now assert the links on the process span and that the invocation span has none.For comparison, the JS SDK's Lambda integration adds no links. It only continues the trace from
ClientContextor headers. Its links are on the SQSReceiveMessagespan only.This is Pablo's code from
pablo/aws-lambda-messaging, minus his link code. On his branch it sat on top of HTTP and invocation attribute work (PY-3017, PY-3019, PY-3021), which isn't part of the November plan. In_handler.pyI only wrapped the handler call in the process span.test_process_span_errorcovers a handler that raises.Open points for review:
messaging.message.idon single-message process spans isn't added yet (proposed in PY-3020).Issues
queue.processspans for messaging triggers #8083via @dingsdax, @sentrivana.
--
View Session [Sentry]