Repository navigation
AWS Lambda failing w/ RuntimeError: can't start new thread on 1.40.3 #2741
Description
How do you use Sentry?
Self-hosted/on-premise
Version
1.40.3
Steps to Reproduce
Recently upgraded from 1.39.2 to 1.40.3.
Our AWS Lambda executors started failing with:
Traceback (most recent call last):
< ... omitted ... >
File "/var/task/sentry_sdk/integrations/threading.py", line 56, in sentry_start
return old_start(self, *a, **kw)
File "/var/lang/lib/python3.9/threading.py", line 899, in start
_start_new_thread(self._bootstrap, ())
RuntimeError: can't start new thread
Rolling back to 1.39.2 prevented the RuntimeError: can't start new thread error.
Expected Result
Lambda execution executed as expected.
Actual Result
A run time exception stemming from the sentry_sdk/threading/threading.py module:
File "/var/task/sentry_sdk/integrations/threading.py", line 56, in sentry_start
return old_start(self, *a, **kw)
File "/var/lang/lib/python3.9/threading.py", line 899, in start
_start_new_thread(self._bootstrap, ())
RuntimeError: can't start new thread
Reacted by Anton Demydov and Chaoying
Metadata
Metadata
Assignees
Projects
- Status
- Status
Activity
sentrivana commentedon Feb 15, 2024
So as said in the original thread I'm thinking this has to do with the SDK spawning a new thread by default in 1.40+ and possibly hitting some thread limits on AWS. We've already seen something similar in this issue, also AWS related.
If my suspicion is correct then this should make it go away:
@thedanfields Could you give this a shot and see if it makes a difference?
gksb88 commentedon Apr 12, 2024
Hi @sentrivana
We upgraded to 1.43.0 and are still seeing this issue with the _experiments={"enable_metrics": False} fix.
The issue happens specifically when a lot of lambdas are writing to s3 or reading from it.
We are on 1.43.0.
kerenkhatiwada commentedon Apr 12, 2024
Hi @sentrivana, can you please also confirm which previous version of the SDK this was working in?
sentrivana commentedon Apr 15, 2024
Hey folks, thanks for following up.
@gksb88 Which SDK version did you upgrade from? Are you creating your own threads in your app?
@kerenkhatiwada According to the OP,
In this case, the most likely culprit from the changes between the two versions (1.39.2 and 1.40.3) was turning metrics on by default, and by extension, starting the background metrics flusher thread. In @gksb88's case though this might be a different issue since turning metrics off doesn't make a difference.
15 remaining items
sentrivana commentedon Jul 26, 2024
Hey @anton-demydov-zoral, SDK 1.x is not developed anymore outside of security fixes. Can you try with the latest 2.x release to see if this is also an issue in 2.x? If yes, can you try my suggestions from above?
See also our migration guide.
lbadi commentedon Oct 15, 2024
We were having the same issue and in our case the cause was a wrong initialization of the sentry sdk.
In our case, the issue was that we were starting the sdk inside the lambda handler function rather than starting it directly on the global context.
In case of a context reutilization, lambda will call multiple times to the sdk with the same context and the threads will start to increase. More information here: https://docs.aws.amazon.com/lambda/latest/dg/lambda-runtime-environment.html
antonpirker commentedon Mar 11, 2025
As this issue is for an old version of the SDK and there are several workarounds and possible solutions for the issue, I will close it.