Skip to content

[gRPC aio] Prior RPC error logs leak into later RPC breadcrumbs despite isolation_scope #8152

Open

Description

@bogdan-zs

How do you use Sentry?

Self-hosted/on-premise. The reproducer captures events in memory and requires no Sentry server.

Version

sentry-sdk 2.68.1; grpcio 1.83.1; protobuf 7.36.1; Python 3.12.

Steps to Reproduce

Install grpcio==1.83.1, protobuf==7.36.1, and sentry-sdk==2.68.1. Save this as issue_repro.py and run python issue_repro.py. It starts a local gRPC server, makes two failing unary-unary calls, and captures Sentry events in memory. The localhost DSN is never contacted.

"""Reproduce cross-RPC breadcrumbs with no Sentry server or generated protobuf files."""

import asyncio
import logging

import grpc
import sentry_sdk
from sentry_sdk.integrations.grpc import GRPCIntegration
from sentry_sdk.integrations.logging import LoggingIntegration
from sentry_sdk.transport import Transport


class MemoryTransport(Transport):
    def __init__(self, options):
        super().__init__(options)
        self.events = []

    def capture_envelope(self, envelope):
        event = envelope.get_event()
        if event and any(
            value.get('mechanism', {}).get('type') == 'grpc'
            for value in event.get('exception', {}).get('values', [])
        ):
            self.events.append(event)


class GrpcLogSignal(logging.Handler):
    def __init__(self):
        super().__init__()
        self.arrived = asyncio.Event()

    def emit(self, record):
        if record.name == 'grpc._cython.cygrpc' and record.getMessage().startswith('Unexpected ['):
            self.arrived.set()


async def fail(request, context):
    sentry_sdk.add_breadcrumb(message=f'rpc:{request.decode()}')
    raise RuntimeError(f'rpc:{request.decode()}')


async def main():
    sink = MemoryTransport({'dsn': 'http://public@localhost/1'})
    sentry_sdk.init(
        dsn='http://public@localhost/1',
        transport=sink,
        default_integrations=False,
        integrations=[
            GRPCIntegration(),
            LoggingIntegration(level=logging.INFO, event_level=logging.ERROR),
        ],
    )
    signal = GrpcLogSignal()
    logger = logging.getLogger('grpc._cython.cygrpc')
    logger.addHandler(signal)
    server = grpc.aio.server()
    server.add_generic_rpc_handlers((
        grpc.method_handlers_generic_handler('repro.Service', {
            'Fail': grpc.unary_unary_rpc_method_handler(
                fail, request_deserializer=lambda data: data,
                response_serializer=lambda data: data,
            ),
        }),
    ))
    port = server.add_insecure_port('127.0.0.1:0')
    await server.start()
    try:
        async with grpc.aio.insecure_channel(f'127.0.0.1:{port}') as channel:
            call = channel.unary_unary('/repro.Service/Fail')
            for number in (1, 2):
                signal.arrived.clear()
                try:
                    await call(str(number).encode())
                except grpc.aio.AioRpcError:
                    pass
                await asyncio.wait_for(signal.arrived.wait(), 5)
                event = sink.events[number - 1]
                messages = [b['message'] for b in event.get('breadcrumbs', {}).get('values', [])]
                print(f'RPC {number} event breadcrumbs: {messages}')
    finally:
        await server.stop(0)
        logger.removeHandler(signal)
        sentry_sdk.get_client().close()

    second = [b['message'] for b in sink.events[1]['breadcrumbs']['values']]
    assert any('Unexpected [RuntimeError]' in message for message in second)
    print('Reproduced: second RPC inherited first RPC error log')


if __name__ == '__main__':
    asyncio.run(main())

Expected Result

The Sentry error event for RPC 2 should contain only breadcrumbs generated during RPC 2. It must not contain the gRPC error log from RPC 1. gRPC can still log the exception normally.

Actual Result

Output from the script above with the pinned versions:

RPC 1 event breadcrumbs: ['rpc:1']
RPC 2 event breadcrumbs: ['Unexpected [RuntimeError] raised by servicer method [/repro.Service/Fail]', 'rpc:2']
Reproduced: second RPC inherited first RPC error log

This differs from #5894, which fixed the absence of an async gRPC isolation_scope() in 2.59.0. In 2.68.1, Sentry captures the exception inside its per-handler scope and re-raises. After that scope exits, gRPC's outer _handle_exceptions logs Unexpected [...]. LoggingIntegration adds the record to the restored outer scope. A subsequent request forks that scope and inherits the breadcrumb.

A local application workaround clears the active Sentry request scope at the start of the RPC handler. It prevents the stale log from reaching the next RPC event, but also removes legitimate breadcrumbs added by middleware earlier in the current RPC. Is there a way for the integration to isolate the entire gRPC RPC task, including the framework's post-handler exception log, or to start the request scope without inherited breadcrumbs?

Activity

  1. linear-code commented on Oct 29, 2026

    @linear-code
  2. AbdulkadirKhoujja commented on Oct 31, 2026

    @AbdulkadirKhoujja

    Hi maintainers,

    I independently reproduced #8152 using Python 3.13.5, sentry-sdk 2.61.0 and grpcio 1.80.0.

    With two sequential failing unary-unary RPCs, the second captured event inherits the framework-generated error breadcrumb from the first request.

    Additional instrumentation suggests that the handler's isolation scope exits before gRPC emits its exception log, allowing that log into an outer scope subsequently inherited by another request.

    I'd be happy to contribute a focused regression test and fix.

    Before implementing, would you prefer a solution that isolates the entire RPC lifecycle or a narrower change to prevent cross-request contamination from post-handler gRPC error logs?

    I'll personally review and test any AI-assisted changes in accordance with your contribution guidelines.

    Thanks!

  3. 18Abhinav07 commented on Oct 31, 2026

    @18Abhinav07

    I had been working on this issue and validating it in my local setup, and I didn't want to comment until that validation was done. When I came back to the issue, I saw the comment above, so I checked those details against current master (2.73.0). The reproduction and cause hold, but the problem turned out to be wider than the post-handler log, which matters for choosing between the two options:

    • The log lands on the isolation scope shared by all RPCs, so the breadcrumbs accumulate rather than affecting only the next request.
    • Async streaming handlers have no isolation_scope() at all. Breadcrumbs added inside unary-stream, stream-unary and stream-stream handlers reach later RPCs' events. A fix limited to post-handler logs would leave this in place.
    • Ignoring grpc._cython.cygrpc isn't an option, since that log is currently the only report of streaming handler failures.

    gRPC runs _handle_rpc inside the _handle_exceptions task via await rpc_coro, so the interceptor and the log share one task. Forking the current and isolation scopes at the top of intercept_service, without restoring them, covers the whole RPC for all four handler types, and the scopes end with the task.

    I have this working locally. A parametrized regression test across the four RPC types fails before the change and passes after it, and the gRPC suite passes on grpcio 1.66.2, 1.85.0 and latest. 4 concurrent streaming RPCs stayed isolated and tags set at startup were preserved. I couldn't build 1.32.0 or 1.49.1 locally, though their source has the same structure.

    One note for testing: clean_scopes resets the isolation scope to None, which hides this after the first test in a run, so the test needs the scope to exist before the server starts.

    If the maintainers are happy with this, I'm glad to open a draft PR, or to take it in a different direction, whichever you prefer.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions