Skip to content

feat(aws-lambda): Add queue.process spans for messaging triggers - #8223

Merged
sentrivana merged 39 commits into
major/3.0from
junior/aws-lambda-queue-process
Nov 12, 2026
Merged

sentrivana merged 39 commits into
major/3.0from
junior/aws-lambda-queue-process

Conversation

@sentry-junior

@sentry-junior sentry-junior Bot commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

Description

For an SQS or SNS delivery, the handler now runs inside a queue.process child span of kind consumer. The span has:

  • messaging.system
  • messaging.operation.type and messaging.operation.name, both process
  • the batch message count
  • the destination name (queue or topic, from the first record)

This fills in the processing side of the Queues insight.

Producer links move to this span. The OTel messaging conventions put a consumer's links to each message's creation context on its process span. #8257 (merged) put them on the invocation span. This PR moves them: the invocation span has no links anymore, and queue.process gets them. The links are built by the same _get_producer_links from #8257 (dedupe, the cap of 128, skipping the continued trace), with no second copy of that parsing. The tests reuse the SQS_RECORD / SNS_RECORD fixtures and the test_producer_links cases already on major/3.0. Those tests now assert the links on the process span and that the invocation span has none.

For comparison, the JS SDK's Lambda integration adds no links. It only continues the trace from ClientContext or headers. Its links are on the SQS ReceiveMessage span only.

This is Pablo's code from pablo/aws-lambda-messaging, minus his link code. On his branch it sat on top of HTTP and invocation attribute work (PY-3017, PY-3019, PY-3021), which isn't part of the November plan. In _handler.py I only wrapped the handler call in the process span.

test_process_span_error covers a handler that raises.

Open points for review:

  • messaging.message.id on single-message process spans isn't added yet (proposed in PY-3020).
  • The destination name comes from the first record. The alternative is to set it only when all records share it.

Issues

via @dingsdax, @sentrivana.

--

View Session [Sentry]

pabloDeputter and others added 8 commits October 23, 2026 15:08
A synchronous boto3 Invoke now carries sentry-trace and baggage in
ClientContext.custom (added with the Lambda service extension). The
Lambda integration continues the trace from there. ClientContext wins
over event headers, because a forwarded HTTP event can keep headers
from further upstream.

The ClientContextTrace test function works around sam local, which
drops ClientContext, by handing the event's client_context to the
runtime.

Co-Authored-By: Johannes Daxböck <114897+dingsdax@users.noreply.github.com>
Classify the delivered event into faas.trigger: http (API Gateway,
function URLs, ALB, CloudFront, AppSync), pubsub (SQS, SNS, Kinesis,
Kafka, MQ, SES, EventBridge), datasource (S3, DynamoDB Streams,
DocumentDB), timer (EventBridge schedules) or other. The trigger is
read from the event envelope, never from its payload.

messaging.batch.message_count is now only set for messaging
deliveries, with the number of delivered messages.

Co-Authored-By: Johannes Daxböck <114897+dingsdax@users.noreply.github.com>
For an SQS, SNS, Kafka, ActiveMQ or RabbitMQ delivery, the handler runs
inside a queue.process child span (kind consumer) with messaging.system,
messaging.operation.type and messaging.operation.name set to process,
the batch message count, and the destination name when all records
share it.

The span links to each distinct producer context found in the records'
Sentry message attributes: SQS messageAttributes, SNS MessageAttributes,
and SQS bodies that wrap an SNS notification.

Co-Authored-By: Johannes Daxböck <114897+dingsdax@users.noreply.github.com>
@linear-code

linear-code Bot commented Nov 9, 2026

Copy link
Copy Markdown

PY-3020

@github-actions

github-actions Bot commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

Coverage Results 📊

✅ 58870 passed | ⏭️ 3071 skipped | Total: 61941 | Pass Rate: 95.04% | Execution Time: 158m 56s

📊 Comparison with Base Branch

Metric Change
Total Tests 📉 -85142
Passed Tests 📉 -80351
Failed Tests 📉 -1
Skipped Tests 📉 -4790

➖ Removed Tests (1)

View removed tests
  • test_openfeature_integration_threaded
    • File: tests.integrations.openfeature.test_openfeature

All tests are passing successfully.

❌ Patch coverage is 12.50% (2 of 16 changed executable lines covered; target 80%).
Project statement coverage is 90.12% (down 0.09 percentage points from base (4213a82) to head (1a11d5a)).

Changed files with executable lines (1)
File Patch coverage Changed executable lines
sentry_sdk/integrations/aws_lambda/_handler.py 12.50% 2/16 covered; missed: 292, 293, 294, 295, 296, 298, 299, 301, 305, 307, 312, 313, 333, 334
Coverage diff
@@            Coverage Diff             @@
##        master     #8223       +/-##
==========================================
- Coverage    90.21%    90.12%    -0.09%
==========================================
  Files          203       200        -3
  Tracked lines     27335     21872     -5463
  Branches     10246      7628     -2618
==========================================
- Hits         24660     19711     -4949
+ Misses        2675      2161      -514
+ Partials      1559      1255      -304

Generated by Coverage Action

sentry-junior Bot and others added 3 commits November 9, 2026 07:57
…a span

Keep the count on messaging spans only (boto3 receive and batch-send
spans, and queue.process for SQS/SNS triggers), as in the OTel
messaging conventions and the JS SDK. This drops the batch counting for
Kafka, MQ and RabbitMQ deliveries, and matches #8219 so both PRs change
these lines the same way.

Co-Authored-By: johdax <114897+dingsdax@users.noreply.github.com>
…-process

Co-Authored-By: johdax <114897+dingsdax@users.noreply.github.com>

# Conflicts:
#	sentry_sdk/integrations/aws_lambda/_handler.py
#	sentry_sdk/integrations/aws_lambda/_trigger.py
The error is captured, both the process span and the invocation span
are marked as errors, and the exception still reaches the Lambda
runtime.

Co-Authored-By: johdax <114897+dingsdax@users.noreply.github.com>
sentry-junior Bot and others added 5 commits November 9, 2026 12:17
Read client_context and its custom dict with getattr, so a context
object without them falls back to the event headers.

Co-Authored-By: Johannes Daxböck <114897+dingsdax@users.noreply.github.com>
…r/aws-lambda-trigger

Co-Authored-By: Johannes Daxböck <114897+dingsdax@users.noreply.github.com>
Each case is a SAM invoke. The removed cases only checked other keys of
the same lookup table.

Co-Authored-By: Johannes Daxböck <114897+dingsdax@users.noreply.github.com>
…-process

Co-Authored-By: Johannes Daxböck <114897+dingsdax@users.noreply.github.com>
OTel sets faas.trigger on the FaaS span, which already has it. Also make
the producer link cap private, like the other module constants.

Co-Authored-By: Johannes Daxböck <114897+dingsdax@users.noreply.github.com>
…a-invoke-trace-continuation

# Conflicts:
#	sentry_sdk/integrations/boto3/_client.py
#	sentry_sdk/integrations/boto3/_instrumentation.py
#	sentry_sdk/integrations/boto3/_services/base.py
#	sentry_sdk/integrations/boto3/_services/lambda_.py
#	sentry_sdk/integrations/boto3/_services/registry.py
@sentry-junior
sentry-junior Bot marked this pull request as ready for review November 11, 2026 17:47
@sentry-junior
sentry-junior Bot requested a review from a team as a code owner November 11, 2026 17:47
sentrivana added a commit that referenced this pull request Nov 12, 2026
)

### Description

Since #8137, #8142, and #8145, boto3 puts `sentry-trace` in SQS and SNS
message attributes, but nothing reads it on the Lambda side. This PR is
the smallest change that closes the loop for SQS- and SNS-triggered
functions. The Lambda integration now reads `sentry-trace` from each
record and adds a `message_origin` link from the invocation span to each
distinct producer span. This matches what the boto3 `ReceiveMessage`
span already does. Messages never become the invocation's parent.

It reads:
- SQS records: `messageAttributes["sentry-trace"].stringValue`
- SNS records: `Sns.MessageAttributes["sentry-trace"].Value`
- SNS → SQS without raw message delivery: the SNS envelope in the SQS
`body`, the same as boto3 does

Links are deduplicated, because a batch send gives many messages the
same producer span, and capped at 128, OTel's default, because Lambda
can deliver up to 10,000 SQS messages at once.

**Scope.** This is intentionally kept separate from the stretch stack.
It doesn't add `faas.trigger` (#8222) or `queue.process` spans (#8223),
and it doesn't touch `messaging.batch.message_count` (#8229). The
invocation span is otherwise unchanged. Synchronous `Invoke` propagation
through `ClientContext` is #8221. That PR changes the `continue_trace`
call and this one only adds `links=`, so the two are independent.

The link-building code repeats boto3's `_message_link` in a Lambda-event
shape: Lambda records use `messageAttributes` / `stringValue` / `body`,
while the boto3 response uses `MessageAttributes` / `StringValue` /
`Body`. Importing from the boto3 integration would make `aws_lambda`
depend on botocore.

**Testing.** `test_producer_links` sends SQS, SNS, and SNS-to-SQS events
to `BasicOkSpanStreaming`. It asserts the segment span's `links` and
that the segment span has no parent. It passes in CI's Cloud (Lambda)
jobs, and `ruff` and `mypy sentry_sdk` pass. `codecov/patch` is red
because the Lambda tests run in `sam local` containers, so their
coverage isn't counted.

#### Issues
* Part of #8081

<!-- junior-request-attribution:start -->
via **webb**, @sentrivana.
<!-- junior-request-attribution:end -->

<!-- junior-session-footer:start -->
<!-- junior-conversation-id:slack%3AC09M26BC6A1%3A1794211222.463239 -->

--

[View
Session](https://junior-prod.sentry.dev/conversations/slack%3AC09M26BC6A1%3A1794211222.463239)
[[Sentry]](https://sentry.sentry.io/explore/conversations/slack%3AC09M26BC6A1%3A1794211222.463239/?project=4510944073809921)

<!-- junior-session-footer:end -->

---------

Co-authored-by: sentry-junior[bot] <264270552+sentry-junior[bot]@users.noreply.github.com>
Co-authored-by: Alexander Webb <alexander.webb@sentry.io>
Co-authored-by: Ivana Kellyer <131587164+sentrivana@users.noreply.github.com>
Restores the batch count on the invocation span, as on major/3.0, so this
PR only adds faas.trigger.
sentrivana pushed a commit that referenced this pull request Nov 12, 2026
… span (#8229)

### Description

The Lambda span set `messaging.batch.message_count` to `1` for every
event that isn't a list, and to `len(event)` for a top-level list. An
SQS or SNS delivery is a dict with `Records`, so a batch of 10 messages
reported 1. HTTP calls and direct invocations reported a batch they
don't have.

This PR removes the attribute from the Lambda span. The count belongs on
messaging spans, as in the OTel messaging conventions:
- boto3 `ReceiveMessage` and batch-send spans already set it (#8137,
#8142).
- For SQS/SNS-triggered Lambdas, the `queue.process` span in #8223 sets
it.

The JS SDK doesn't set it on the Lambda span either. The attribute was
added in 3.0 (#7941) and never shipped in 2.x, so no released behavior
changes.

Replaces #8219 (same commits). GitHub never ran CI on #8219.

**Note for review:** this is narrower than PY-3015, which asks to keep
the attribute for SQS/SNS deliveries. @alexander-alderman-webb, please
confirm that's OK.

#### Issues
* resolves: PY-3015

<!-- junior-request-attribution:start -->
via @dingsdax.
<!-- junior-request-attribution:end -->

<!-- junior-session-footer:start -->
<!-- junior-conversation-id:slack%3AC0C9X9V5NMT%3A1793904671.980379 -->

--

[View
Session](https://junior-prod.sentry.dev/conversations/slack%3AC0C9X9V5NMT%3A1793904671.980379)
[[Sentry]](https://sentry.sentry.io/explore/conversations/slack%3AC0C9X9V5NMT%3A1793904671.980379/?project=4510944073809921)

<!-- junior-session-footer:end -->

---------

Co-authored-by: sentry-junior[bot] <264270552+sentry-junior[bot]@users.noreply.github.com>
Co-authored-by: Johannes Daxböck <114897+dingsdax@users.noreply.github.com>

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 8aefb75. Configure here.

sentrivana pushed a commit that referenced this pull request Nov 12, 2026
### Description

Sets `faas.trigger` on the Lambda span based on the shape of the
delivered event:
- `http`: API Gateway REST/HTTP, function URLs, ALB, CloudFront, AppSync
- `pubsub`: SQS, SNS, Kinesis, Kafka, MQ, SES, EventBridge
- `datasource`: S3, DynamoDB Streams, DocumentDB
- `timer`: EventBridge schedules
- `other`: anything else

The trigger is read from the event envelope, never from its payload.
`test_trigger` has one case per code path. Other event sources use the
same lookup table, so they have no extra cases.

This PR doesn't touch `messaging.batch.message_count`. That change is in
#8229.

This is Pablo's code from `pablo/aws-lambda-trigger`, without his batch
counting for the invocation span. It's the second of three stacked PRs
(#8221 → this → #8223).

#### Issues
* resolves: PY-3018 (`faas.trigger` part; the per-source attributes in
the issue aren't in this PR)
* Part of #8081

<!-- junior-request-attribution:start -->
via @dingsdax.
<!-- junior-request-attribution:end -->

<!-- junior-session-footer:start -->
<!-- junior-conversation-id:slack%3AC0C9X9V5NMT%3A1793904671.980379 -->

--

[View
Session](https://junior-prod.sentry.dev/conversations/slack%3AC0C9X9V5NMT%3A1793904671.980379)
[[Sentry]](https://sentry.sentry.io/explore/conversations/slack%3AC0C9X9V5NMT%3A1793904671.980379/?project=4510944073809921)

<!-- junior-session-footer:end -->

---------

Co-authored-by: Pablo Deputter <pablo.deputter@sentry.io>
Co-authored-by: Pablo Deputter <71842639+pabloDeputter@users.noreply.github.com>
Co-authored-by: sentry-junior[bot] <264270552+sentry-junior[bot]@users.noreply.github.com>
Co-authored-by: Johannes Daxböck <114897+dingsdax@users.noreply.github.com>
Base automatically changed from junior/aws-lambda-trigger to major/3.0 November 12, 2026 13:47
@sentrivana

Copy link
Copy Markdown
Collaborator

@sentry-junior Update this branch with latest major/3.0 and solve any merge conflicts

…a-queue-process

# Conflicts:
#	sentry_sdk/integrations/aws_lambda/_handler.py
#	sentry_sdk/integrations/aws_lambda/_trigger.py
#	tests/integrations/aws_lambda/test_aws_lambda.py
@sentry-junior

sentry-junior Bot commented Nov 12, 2026

Copy link
Copy Markdown
Contributor Author

Done. I merged major/3.0 into this branch in a922aa6, right after #8222 landed. There are no newer commits on major/3.0, and GitHub shows the PR as mergeable.

The conflicts were in _handler.py, _trigger.py and test_aws_lambda.py. In all three, major/3.0 now has exactly #8222's final version, and this branch already had it, so I kept this branch's side. Nothing from #8222 or major/3.0 was dropped. ruff and mypy pass, and CI is running.

sentry-junior Bot and others added 2 commits November 12, 2026 13:56
#8257 links SQS/SNS invocations to their producers on the invocation
span. Remove this PR's second copy of that code from _trigger.py, keep the
queue.process span without links, and reuse the test helpers and records
already on major/3.0.

Co-Authored-By: Ivana Kellyer <131587164+sentrivana@users.noreply.github.com>
Following the OTel messaging conventions, the process span carries the
links to each message's creation context. Move the links that #8257 put on
the invocation span to the queue.process span, reusing the same
_get_producer_links.

Co-Authored-By: Ivana Kellyer <131587164+sentrivana@users.noreply.github.com>
@sentrivana

Copy link
Copy Markdown
Collaborator

Does this need a migration guide update?

@sentry-junior
sentry-junior Bot force-pushed the junior/aws-lambda-queue-process branch from 0cb05d4 to 1a11d5a Compare November 12, 2026 14:12
@sentry-junior

sentry-junior Bot commented Nov 12, 2026

Copy link
Copy Markdown
Contributor Author

I don't think it needs one. Everything this PR adds is new, and nothing that 2.x users rely on is renamed or removed:

The one visible difference is that spans a user starts inside an SQS- or SNS-triggered handler now have queue.process as their parent, not the function span. That's the usual result of adding an instrumentation span, and the guide doesn't list that kind of change for other integrations. I'd leave it out unless you want it called out.

@sentrivana
sentrivana merged commit 90c4eec into major/3.0 Nov 12, 2026
152 checks passed
@sentrivana
sentrivana deleted the junior/aws-lambda-queue-process branch November 12, 2026 14:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants