We want to improve the data we collect for task queues.
We will start with enhancing our Celery integration. This should work with redis, and rabbitmq, and as a stretch goal amazon sqs as brokers. (Kafka should follow later.)
How spans are right now:
- Producer creates
<span op="queue.submit.celery"> when task is put into queue. (start_timestamp is when putting into queue is started and end_timestamp is when the item is in the queue.)
- Consumer creates
<transaction op="queue.task.celery"> when task is fetched and processed. (start_timestamp is when the item was received from the queue, end_timestamp is when the processing of the task has finished. )
- Transaction has a status:
ok in case of success
aborted if Celery raises Retry, Ignore or Reject celery exceptions
internal_error if an unhandled exception was raised during execution of the task
New data we want to add to queue spans:
(for naming see: https://opentelemetry.io/docs/specs/semconv/attributes-registry/messaging/)
- retry count (can be found in
args of existing _wrap_tracer, the attached PR)
- byte size of message (
messaging.message.body.size)
- byte size of message plus metadata (
messaging.message.envelope.size)
- success/failure status - already there
- host (
server.address) - (can be found in args of existing _wrap_tracer, the attached PR)
- queue or topic (
messaging.destination.name) - not sure where to get this.
- message id (
messaging.message.id) - should also be in args of the comments in the attached PR)
- consumer id (unique id of the consumer) (
messaging.client_id) - dont know yet where to get.
- what broker is used in celery. (
messaging.system)
See this PR for some points where to get the data: #2962
I also added a sample project that uses RabbitMQ with Celery: https://github.com/antonpirker/testing-sentry/tree/main/test-celery-queues-module
Mock of how it will look in the product:

### Tasks
- [ ] https://github.com/getsentry/sentry-python/issues/2959
We want to improve the data we collect for task queues.
We will start with enhancing our Celery integration. This should work with redis, and rabbitmq, and as a stretch goal amazon sqs as brokers. (Kafka should follow later.)
How spans are right now:
<span op="queue.submit.celery">when task is put into queue. (start_timestampis when putting into queue is started andend_timestampis when the item is in the queue.)<transaction op="queue.task.celery">when task is fetched and processed. (start_timestampis when the item was received from the queue,end_timestampis when the processing of the task has finished. )okin case of successabortedif Celery raises Retry, Ignore or Reject celery exceptionsinternal_errorif an unhandled exception was raised during execution of the taskNew data we want to add to queue spans:
(for naming see: https://opentelemetry.io/docs/specs/semconv/attributes-registry/messaging/)
argsof existing_wrap_tracer, the attached PR)messaging.message.body.size)messaging.message.envelope.size)server.address) - (can be found inargsof existing_wrap_tracer, the attached PR)messaging.destination.name) - not sure where to get this.messaging.message.id) - should also be inargsof the comments in the attached PR)messaging.client_id) - dont know yet where to get.messaging.system)See this PR for some points where to get the data: #2962
I also added a sample project that uses RabbitMQ with Celery: https://github.com/antonpirker/testing-sentry/tree/main/test-celery-queues-module
Mock of how it will look in the product: