uWSGI workers dying in 1.40.0

How do you use Sentry?

Sentry Saas (sentry.io)

Version

1.40.0

Steps to Reproduce

I don't have a reproduction yet due to the nature of the issue, for which I apologize, but with 1.40.0 my server processes are silently dying after about 90 minutes of run time. This does not happen with 1.39.2. I just want to get it on your radar that the latest update may be causing issues for some. I wish I had more information, but I don't at this time.

Expected Result

^

Actual Result

^

I also had a similar issue with 1.40, it was killing my uwsgi workers, had to downgrade to 1.39. Also wish I had more information to give but there wasn't much to go on. The issue was worse with profiling on (workers would be killed via SIGBUS)

@fiendish

@fiendish @seb-b Hey folks, thanks for bringing this to our attention. Even if there is no repro, more info could help point us in the right direction -- what kind of apps (Django, Flask, FastAPI, something else?) are you running? Which SDK integrations are active (initing the SDK with debug=True should log this at startup)?

Looking at the changelog from 1.40.0 there's a couple things we could try:

turning off metrics collection:

sentry_sdk.init(
    ... # your usual stuff
    _experiments={
        "enable_metrics": False,
    }
)

turning off DB query source capture:

sentry_sdk.init(
    ... # your usual stuff
    enable_db_query_source=False,
)

Oops, yeah, sorry. In my haste I forgot to include. In my case it's Flask 2.1.3 + uWSGI 2.0.23, and

integrations = [
    FlaskIntegration(),
    LoggingIntegration(
        level=logging.INFO,
        event_level=logging.ERROR,
    )
]

I'll try to find time over the weekend or monday to try those things.

Same here with Django 3.2 LTS, uWSGI 2.0.23. This are my settings:

sentry_logging = LoggingIntegration(
    level=logging.INFO,  # Capture info and above as breadcrumbs
    event_level=logging.ERROR,  # Send errors as events
)
sentry_settings = {
    "dsn": SENTRY_LIVE_DSN,
    "integrations": [DjangoIntegration(), sentry_logging],
    "send_default_pii": True,
    "enable_tracing": True,
}

@PAStheLoD

I can reproduce the issue.

uwsgi invocation with a.py from @PAStheLoD:

uwsgi -w a:flask_app --need-app --http-socket 127.0.0.1:8080 -p 4 --enable-threads

The segfault is caused by uwsgi not handling threads correctly with default arguments.

When forking python with multiple threads running the fork hooks PyOS_BeforeFork, PyOS_AfterFork_Child and PyOS_AfterFork_Parent have to be run before/after fork. A correct fork sequence looks something like this. uwsgi runs none of those by default. It seems like this is by design, as uwsgi assumes that wsgi apps are single threaded (source).

To make the segfault disappear uwsgi has to be called with --enable-threads (initialize the GIL) and --py-call-uwsgi-fork-hooks (run PyOS_AfterFork_Child in the worker after fork).

uwsgi -w a:flask_app --need-app --http-socket 127.0.0.1:8080 -p 4 --enable-threads --py-call-uwsgi-fork-hooks

Interestingly this behaviour is still not fully correct: Only PyOS_AfterForkChild is called. PyOS_BeforeFork and PyOS_AfterFork_Parent are not called. I'm not sure if that can lead to segfaults too, but in any case not all hooks registered by os.register_at_fork are called when the worker is forked. (The before and after_in_parent hooks are not called.)

Not sure if there is any viable workaround for the sentry SDK, as the corruption happens outside of its control.

IMHO this is not a sentry issue, but caused by unsafe defaults in uwsgi. I wish uwsgi would pick safer defaults.

Used software versions:
Linux 6.1.71-1-lts (arch linux kernel build)
Python 3.11.8 (arch linux PKGBUILD, but rebuilt with debug symbols)
uwsgi 2.0.23 and 2.0.24
sentry_sdk 1.40.2

@natano

Thanks a lot for looking into this @natano!

I think the best we can do on the SDK side moving forward is to issue a warning on startup if we detect we're in non-lazy mode and --py-call-uwsgi-fork-hooks is not on, just like we do currently if you're running uWSGI without --enable-threads.

The fix in 1.40.4 should work even without --py-call-uwsgi-fork-hooks in most cases because with that release we spawn all threads on demand BUT if you're unlucky and a thread is spawned at startup before forking, you might run into this again.

So to sum up for future reference, if you run into this issue:

run uWSGI with both --enable-threads and --py-call-uwsgi-fork-hooks
OR run uWSGI with --lazy-apps (or --lazy, but this is discouraged) and --enable-threads

To add some extra seasoning to this issue.

We saw failures in our AWS Lambda executors after moving to 1.40.3:

Traceback (most recent call last):
  < ... omitted ... >
  File "/var/task/sentry_sdk/integrations/threading.py", line 56, in sentry_start
    return old_start(self, *a, **kw)
  File "/var/lang/lib/python3.9/threading.py", line 899, in start
    _start_new_thread(self._bootstrap, ())
RuntimeError: can't start new thread

@thedanfields

@thedanfields I'd wager a guess that's a AWS specific issue - reminds me of #2632 without the upgrading to Python 3.12 part. I wonder if we've hit some kind of AWS Lambda limit on the number of threads since the SDK is now spawning one additional thread.

Could you open a new issue with this please?

@sentrivana

@sentrivana , happy to oblige

I created a new issue as requested.

fix: Upgrade SDK and reenable DDM

fix: Upgrade SDK and reenable DDM (#5557)

Navigation Menu

Repository navigation

uWSGI workers dying in 1.40.0 #2699

How do you use Sentry?

Version

Steps to Reproduce

Expected Result

Actual Result

67 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

uWSGI workers dying in 1.40.0 #2699

Description

How do you use Sentry?

Version

Steps to Reproduce

Expected Result

Actual Result

Activity

seb-b commented on Feb 2, 2024

sentrivana commented on Feb 2, 2024

fiendish commented on Feb 2, 2024

gerricom commented on Feb 2, 2024

67 remaining items

natano commented on Feb 14, 2024

sentrivana commented on Feb 14, 2024

thedanfields commented on Feb 15, 2024

sentrivana commented on Feb 15, 2024

thedanfields commented on Feb 15, 2024

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions