__import__ is not thread-safe on Python 3

@brettcannon

BPO	38884
Nosy	@brettcannon, @gpshead, @ncoghlan, @ericsnowcurrently, @mulka
Files	import_module_not_found.py import_deadlock.py issue38884.zip

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2019-11-21.19:11:54.536>
labels = ['3.7', '3.8', '3.9']
title = '__import__ is not thread-safe on Python 3'
updated_at = <Date 2021-12-08.22:53:27.704>
user = 'https://bugs.python.org/ValentynTymofieiev'

bugs.python.org fields:

activity = <Date 2021-12-08.22:53:27.704>
actor = 'Valentyn Tymofieiev'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = []
creation = <Date 2019-11-21.19:11:54.536>
creator = 'Valentyn Tymofieiev'
dependencies = []
files = ['48734', '48735', '48737']
hgrepos = []
issue_num = 38884
keywords = []
message_count = 14.0
messages = ['357198', '357199', '357202', '357214', '357227', '357228', '357230', '357234', '357464', '360068', '360069', '362746', '375958', '408052']
nosy_count = 9.0
nosy_names = ['brett.cannon', 'gregory.p.smith', 'gjb1002', 'ncoghlan', 'eric.snow', 'ryan.petrello', 'Valentyn Tymofieiev', 'Martin Laus', 'repalviglator']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = None
url = 'https://bugs.python.org/issue38884'
versions = ['Python 3.7', 'Python 3.8', 'Python 3.9']

Linked PRs

GH-83065: Fix import deadlock by implementing hierarchical module locking #137196

Attached import_module_not_found.py consistently fails for me on Python 3.7.5 and earlier Python 3 versions that I have tried with

File "/usr/lib/python3.7/threading.py", line 926, in _bootstrap_inner
self.run()
File "/usr/lib/python3.7/threading.py", line 870, in run
self._target(*self._args, **self._kwargs)
File "import_module_not_found.py", line 8, in t2
return __import__("tensorflow.estimator", level=0)
ModuleNotFoundError: No module named 'tensorflow.estimator

Threads in this example finish successfully if executed sequentially.

I have not tried higher versions of Python, but I cannot reproduce this on Python 2.7.

Is this an expected behavior?

Thank you.

Attached import_deadlock.py fails with

...
File "<frozen importlib._bootstrap>", line 980, in _find_and_load
File "<frozen importlib._bootstrap>", line 149, in __enter__
File "<frozen importlib._bootstrap>", line 94, in acquire
_frozen_importlib._DeadlockError: deadlock detected by _ModuleLock('tensorflow_transform.tf_metadata.metadata_io')

under the same conditions.

Possibly related: https://bugs.python.org/issue35943

Can you test this with a package that isn't tensorflow (e.g. something else in the stdlib or something created manually; basically a smaller reproducer)? It's hard to diagnose if this is really Python or not without a smaller reproducer as tensorflow might be doing some things behind the scenes that would break this due to it being an extension module.

Attaching a minimal repro with simple project hierarchy.

Repro: unpack bpo-38884.zip
python input_deadlock.py

(fails on Python 3.7, but not on 2.7).

input_deadlock.py in bpo-38884.zip has a left-over comment:
# Requires pip install tensorflow==2.0.0 tensorflow-transform==0.15.0
Please ignore that.

The behavior changes between Python 3.2 (no deadlock) and Python 3.3 (deadlock). Deadlock also reproducible on Python 3.8.

@brett.cannon, looks like I inadvertently dropped you from nosy list, so in case you missed the updates - please take a look at bpo-38884.zip. Thank you.

Just checking - is bpo-38884.zip an acceptable repro for this issue?
Thanks.

@Valentyn

@Valentyn just assume it's good enough unless some asks otherwise. People are probably too busy right now to be looking at this and so you might be waiting a while for a more definitive answer.

I believe I'm also encountering some version of this bug while importing code from the kombu library within a multi-threaded context.

For what it's worth, I'm able to reproduce the reported deadlock (https://bugs.python.org/file48737/issue38884.zip) using python3.8 on RHEL8 and CentOS 8 builds.

More details about what I'm encountering here: ansible/awx#5617
ansible/awx#5617 (comment)

My intended workaround for now is to just not use a thread (we'll see if it helps):

ansible/awx#6093

The case as I understand it is that T1 imports package.subpackage, which in it's __init__.py imports package.subpackage.module
Simultaneously, T2 is trying to import package.subpackage.module, which should involve importing package.subpackage?

Then you end up with a deadlock where T1 needs to resolve package.subpackage.module to resolve package.subpackage, while T2 needs to resolve package.subpackage to resolve package.subpackage.module, so we end up with a deadlock where they are holding mutually referential locks.

Here's the traceback on main:

Exception in thread Thread-1 (t1):
Traceback (most recent call last):
  File "/home/emma/cpython/Lib/threading.py", line 1074, in _bootstrap_inner
    self._context.run(self.run)
    ~~~~~~~~~~~~~~~~~^^^^^^^^^^
  File "/home/emma/cpython/Lib/threading.py", line 1016, in run
    self._target(*self._args, **self._kwargs)
    ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/emma/test_import_race/import_deadlock.py", line 5, in t1
    return __import__("package.subpackage", level=0)
  File "/home/emma/test_import_race/package/__init__.py", line 1, in <module>
    from package import subpackage
  File "/home/emma/test_import_race/package/subpackage/__init__.py", line 1, in <module>
    from package.subpackage.module import *
  File "<frozen importlib._bootstrap>", line 1368, in _find_and_load
  File "<frozen importlib._bootstrap>", line 421, in __enter__
  File "<frozen importlib._bootstrap>", line 346, in acquire
_frozen_importlib._DeadlockError: deadlock detected by _ModuleLock('package.subpackage.module') at 138200313720016

I'm not really sure what can be done to solve this other than some kind of earlier check when resolving package.subpackage.module if another thread is resolving a parent (in this case package.subpackage) and may need to resolve package.subpackage.module? Then block that thread until the other thread is done.

pythongh-83065: Fix import deadlock by implementing hierarchical modu…

The blunt force workaround for import madness of this form regardless of cause has always been: Do your imports from the main thread. Not super friendly, and extra annoying when it's something nobody likes to think about such as unplanned indirect imports such as those from codecs dynamically loading infrequently used codec implementations, possibly due to untrusted data, in a serving stack.

Out of curiosity, I just had Claude Opus 4 take a stab at this as Emma's analysis sounded right. Draft PR from that attached. If that is sensible (I'll give it more of a look later), I think real performance tests are needed. But hierarchical lock acquisition was also going to be my suggestion w/o having re-read importlib guts as well, so it maybe served as a bit of validation of both the issue & analysis and that thought.

Claude code prompts (/model is opus 4):

load github issue 83065 and analyze the workings of Lib/importlib/ to understand the situation. focus on emmatyping's issue comment, is that accurate given how the importlib code works? what could be done to avoid this?
Plan the import system improvements
before benchmarking, look over the diff so far with the eye of a code reviewer and simplify it. also remove the _use_hierarchical_import_locks logic, we'll always want this enabled. -- (this helped the code quality a lot!)
run the reindent tool -- (it knows how via my CLAUDE.local.md)
commmit your work, credit Emma for the analysis in the commit message.
be sure to put markdown backticks around __init__.py
push to the gpshead remote and create a Draft PR, prefix the title of the PR with "__import__ is not thread-safe on Python 3 #83065: "

half an hour of API use. this cost $$ money (stated just to let others know the scale).

Add NEWS entry and What's New for the pythongh-83065 import deadlock fix

GH-83065: Fix import deadlock by implementing hierarchical module loc…

thanks all for earlier reviews! i found some remaining nits and fixed them, polished up, documented and merged. if we uncover problems with this during 3.15 betas - reverting to the prior status quo can happen. so many things in the world have worked around this long-standing flaw in odd manners. it'll be nice to stop encountering those.

Thanks for fixing this. I have personally made a donation to PSF in your honor.

thanks! 😁

FTR there seem to be also a rare race in stdlib with collections/collections.abc fixed by the linked PR, and causing ModuleNotFoundError: No module named 'collections.abc'

Thread A:
  import collections
    -> _find_and_load("collections")
    -> execute collections/__init__.py
    -> stop between these lines:
        ```
          # collections/__init__.py:29
          import _collections_abc
          import sys
          # INTERRUPTED HERE
          _sys.modules['collections.abc'] = _collections_abc
          abc = _collections_abc
        ```
        -> "collections" is already in sys.modules
        -> although spec._initializing is still True  


Thread B:
  import collections.abc
    -> _find_and_load("collections.abc")
    -> parent "collections" already exists in sys.modules
    -> do not wait for the parent package to finish (no check of _initializing on the spec)     
    -> search collections.__path__ for an abc submodule
    -> no collections/abc.py so no finder result
    -> ModuleNotFoundError at _bootstrap.py:1329

self-contained reproducer fails for me locally on 3.13 and passes on 3.15 dev build

FYI - The fix we landed in 3.15 is not complete and I & claude have discovered problems with it. I'll open a new issue with details and a proposed fix soon.

gh-159963: Don't hold parent package import locks while a submodule e…

[3.15] gh-159963: Don't hold parent package import locks while a subm…

Navigation Menu

Repository navigation

import is not thread-safe on Python 3 #83065

Linked PRs

11 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

__import__ is not thread-safe on Python 3 #83065

Description

Linked PRs

Activity

ValentynTymofieiev commented on Nov 21, 2019

ValentynTymofieiev commented on Nov 21, 2019

ValentynTymofieiev commented on Nov 21, 2019

brettcannon commented on Nov 21, 2019

ValentynTymofieiev commented on Nov 21, 2019

ValentynTymofieiev commented on Nov 21, 2019

ValentynTymofieiev commented on Nov 21, 2019

ValentynTymofieiev commented on Nov 21, 2019

ValentynTymofieiev commented on Nov 25, 2019

ValentynTymofieiev commented on Jan 15, 2020

brettcannon commented on Jan 15, 2020

ryanpetrello commented on Feb 26, 2020

11 remaining items

emmatyping commented on Jul 29, 2025

gpshead commented on Jul 29, 2025

gpshead commented on Apr 28, 2026

bdraco commented on Apr 28, 2026

gpshead commented on May 8, 2026

steve-s commented on Sep 2, 2026

gpshead commented on Oct 13, 2026

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions

import is not thread-safe on Python 3 #83065