Repository navigation
Automatically create monitors when using Redbeat #2616
Description
Environment
SaaS (https://sentry.io/)
Steps to Reproduce
We're using Python with Celery and Redbeat and following the docs for setup. Using either the autodiscover or manual @monitor instrumentation outlined in the docs, we are not having monitors in our Sentry UI automatically created. But manually creating them confirms that the data does come into Sentry—as soon as they're created we can see data for them in the UI, but only from that point forward.
I've been talking to Sentry support about this and was asked to create an Issue. Unfortunately our repo is private but our setup is pretty straightforward. Here's the gist of what we've got:
requirements.txt:
celery==5.2.6
celery-redbeat==2.0.0
worker.py
from celery import Celery
from celery.signals import celeryd_init
from celery.schedules import crontab
import sentry_sdk
from sentry_sdk.crons.decorator import monitor
from sentry_sdk.integrations.celery import CeleryIntegration
from myco import _my_celery_task, get_env_var, LockInUse, get_redis_url
app = Celery(__name__, broker=get_redis_url(database='celery'))
@celeryd_init.connect
def init_sentry(**_kwargs):
env = get_env_var('APP_ENV', False)
if not env:
print('APP_ENV not set, defaulting to development')
env = 'development'
if env not in ['development', 'test']:
sentry_sdk.init(
dsn="<DSN@ingest.sentry.io>", # obviously this is redacted
environment=env,
release=get_env_var('SENTRY_RELEASE_VERSION', False) or 'unknown',
ignore_errors=[LockInUse],
integrations=[CeleryIntegration()],
# Set traces_sample_rate to 1.0 to capture 100%
# of transactions for performance monitoring.
# We recommend adjusting this value in production,
traces_sample_rate=0.005,
)
# ...
@app.task
@monitor(monitor_slug='my_celery_task')
def my_celery_task():
return _my_celery_task()
# ...
app.conf.redbeat_redis_url = get_redis_url(database='celery')
app.conf.beat_scheduler = 'redbeat.RedBeatScheduler'
app.conf.beat_schedule = {
'my-celery-task': {
'task': 'worker.my_celery_task',
'schedule': crontab(minute='0', hour='0', day_of_week='1'),
},
# ...
}Expected Result
Cron job monitors are automatically created in Sentry on deploy, or at least when they execute
Actual Result
No monitors are created
Product Area
Crons
Link
No response
DSN
No response
Version
No response
Metadata
Metadata
Assignees
Projects
- Status
- Status
Activity
getsantry commentedon Dec 20, 2023
Assigning to @getsentry/support for routing ⏲️
getsantry commentedon Dec 20, 2023
Routing to @getsentry/product-owners-crons for triage ⏲️
gaprl commentedon Dec 22, 2023
@antonpirker could you take a look into this please when you're back?
IevgeniiB commentedon Dec 28, 2023
Hi! Just adding my observations because I have been directed to this ticket by support.
I noticed a similar problem - I am using cron monitors for multiple automations that don't use celery. I used to have celery and django integrations enabled by default (because of historical reasons that don't make sense anymore) and decided to disable them for my scripts. It resulted in a better performance on the cron monitors' side.
See the screenshot below:
The gray line is at the point when the integrations were disabled.
As you can see - it significantly improved the stability of crons, but didn't fix the issue entirely. Is there anything else in addition to celery that I should watch out for?
The version of sentry sdk for python that I use is
1.39.1antonpirker commentedon Jan 3, 2024
@IevgeniiB please create a issue in https://github.com/getsentry/sentry-python/ because it is something different the this issue. Thanks!
antonpirker commentedon Jan 3, 2024
@bensaufley looks like task creation does not work when using redbeat. I will have a look.
2 remaining items
antonpirker commentedon Jan 3, 2024
What I know now:
For Celery beat we patch
celery.Beat.Scheduler.apply_entryto add themonitor-configto check-ins. In Redbeats scheduler this function is not used, but instead this: https://github.com/sibson/redbeat/blob/v2.0.0/redbeat/schedulers.py#L450So we need to patch this function to make it work when Redbeats scheduler is used.
IevgeniiB commentedon Jan 3, 2024
@antonpirker thank you! I've submitted a new issue with more info here: #2617
antonpirker commentedon Jan 4, 2024
Ok, we will add support for Redbeat. When it will be done, this issue will be updated of course.
[-]Python Cron Jobs fail to automatically create monitors[/-][+]Automatically create monitors when using Redbeat[/+]