pyclbr.readmodule_ex traversing "import __main__": dies with ValueError: __main__.__spec__ is None / is not set

@kxrob

BPO	43299
Nosy	@kxrob, @akulakov
PRs	gh-87465: Make pyclbr.readmodule_ex() not die on module w/o __spec__ #24623

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2021-02-22.17:48:25.398>
labels = ['type-bug', 'library', '3.10']
title = 'pyclbr.readmodule_ex traversing "import __main__": dies with ValueError: __main__.__spec__ is None / is not set'
updated_at = <Date 2021-07-12.17:42:22.974>
user = 'https://github.com/kxrob'

bugs.python.org fields:

activity = <Date 2021-07-12.17:42:22.974>
actor = 'andrei.avk'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2021-02-22.17:48:25.398>
creator = 'kxrob'
dependencies = []
files = []
hgrepos = []
issue_num = 43299
keywords = ['patch']
message_count = 7.0
messages = ['387520', '397054', '397247', '397279', '397280', '397330', '397342']
nosy_count = 2.0
nosy_names = ['kxrob', 'andrei.avk']
pr_nums = ['24623']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue43299'
versions = ['Python 3.10']

Linked PRs

gh-87465: Bypass sys.modules when finding module specs in pyclbr #161864

When pyclbr.readmodule_ex() is traversing "import __main__" or another
module without __spec__, it dies completely
with "ValueError: __main__.__spec__ is None / is not set".

=> It should at least continue with the (big) rest
as the comment in _ModuleBrowser.visit_Import() says:
# If we can't find or parse the imported module,
# too bad -- don't die here.

And optionally fall back to using __file__ when present?

Traceback (most recent call last):
  File "C:\Python310\Lib\site-packages\pythonwin\pywin\framework\editor\ModuleBrowser.py", line 128, in OnActivateView
    self.CheckRefreshList()
  File "C:\Python310\Lib\site-packages\pythonwin\pywin\framework\editor\ModuleBrowser.py", line 181, in CheckRefreshList
    self.CheckMadeList()
  File "C:\Python310\Lib\site-packages\pythonwin\pywin\framework\editor\ModuleBrowser.py", line 173, in CheckMadeList
    self.rootitem = root = self._MakeRoot()
  File "C:\Python310\Lib\site-packages\pythonwin\pywin\framework\editor\ModuleBrowser.py", line 153, in _MakeRoot
    data = reader(mod, path and [path])
  File "C:\Python310\lib\pyclbr.py", line 120, in readmodule_ex
    return _readmodule(module, path or [])
  File "C:\Python310\lib\pyclbr.py", line 159, in _readmodule
    return _readmodule(submodule, parent['__path__'], package)
  File "C:\Python310\lib\pyclbr.py", line 184, in _readmodule
    return _create_tree(fullmodule, path, fname, source, tree, inpackage)
  File "C:\Python310\lib\pyclbr.py", line 272, in _create_tree
    mbrowser.visit(ast.parse(source))
  File "C:\Python310\lib\ast.py", line 410, in visit
    return visitor(node)
  File "C:\Python310\lib\ast.py", line 418, in generic_visit
    self.visit(item)
  File "C:\Python310\lib\ast.py", line 410, in visit
    return visitor(node)
  File "C:\Python310\lib\pyclbr.py", line 243, in visit_Import
    _readmodule(module.name, [])
  File "C:\Python310\lib\pyclbr.py", line 167, in _readmodule
    spec = importlib.util._find_spec_from_path(fullmodule, search_path)
  File "C:\Python310\lib\importlib\util.py", line 69, in _find_spec_from_path
    raise ValueError('{}.__spec__ is None'.format(name))
ValueError: __main__.__spec__ is None

Robert: I haven't worked with importlib or pyclbr before, so these may be naive questions, but:

can your usecase be resolved with a workaround, e.g. setting ModuleSpec manually in / on __main__ module, something like `from importlib._bootstrap import ModuleSpec; __spec__ = ModuleSpec('main',None)
You mention that this issue may apply to other modules, but the Python docs say __main__ is the only case that may have __spec__=None in some cases (in my testing it indeed have it set to None). Do you have any examples where other modules have __spec__=None?

What's your usecase for examining __main__ with pyclbr? Not trying to sound doubtful, just really curious.

Re your PR I have a bit of a concern that catching a ValueError silently might mask other types of ValueError, but I haven't looked more closely into that, just want to note it here for the future.

Btw thanks for the reply on PR to my question, it makes sense.

You see the usecase from the stack trace: PythonWin (the IDE from pywin32 package) uses pyclbr - to inspect arbitrary user code.
(Neither code is from me)

I'm not inspecting __main__ explicitely. The problem seems to arise in newer Python versions (3.10+?) because the class browser now seems to parse imports somehow recursively (readmodule() several times in the stack trace) and when user code somewhere contains e.g. "import __main_" ...

pyclbr should perhaps handle (not fail in) all legal cases w/o breaking: when some strange builtin/extension/artificial has .__spec__ as None or not set or no python code. (We cannot force any user code to monkey patch __main__.__spec__ or potential other modules.)

>>> mod = types.ModuleType('mymod')
>>> mod.__spec__
# (None)
>>>

importlib.util._find_spec_from_path() has choosen to raise ValueError instead of an extra custom Error (or a special return value) for those cases and to return None for the similar case of 'not found') . Though those 3 cases are similiar in consequence here. pyclbr also "cheaply abuses" ImportError / ModuleNotFound to translate one of those cases (None return value) for internal signaling. (There is never a real ImportError just remote linguistic similarity ;-) )

Hence the simple pragmatic fix by kind of reunification of signaling the "end of the game" here - as the way of signaling here is anyway rather pragmatic and evolution style.

ValueError is often (ab)used to signal application level errors "cheaply" (to not define and distribute an extra Exception type) - and its a limited internal case here where mix-up with errors from something like "math.sqrt(-1)" is not possible w/o coding bugs (which are to be detected by tests)

But you may establish a more meticulous error / return value signaling system - which though will require changes / refactoring in several places and consideration for compatibility ...
(Think its hardly worth it)

Robert: thanks for the response..

I looked a bit more into it and I'm starting to think it's a PyWin issue, not pyclbr.

It seems to me that PyWin is calling pyclbr readmodule_ex with module='__main__', and pyclbr is supposed to find it on the filesystem. The problem is, '__main__' is the special name for current script or interactive session. It doesn't give pyclbr any info on where the script might be, if it exists (maybe it's pywin's interactive window?).

So, unless someone more familiar with pyclbr steps in and gives us a bit more insight, I think the best course may be to report this to PyWin. They might also know more about pyclbr, perhaps they've run into similar issues with it before!

(Also if they are able to help, please report back here and update this issue, that will be much appreciated!)

Actually, I was mistaken about this, readmodule_ex can get the main from sys.modules.. I will look more into this later today.

Upon further pondering, I'm not sure PyWin is doing the right thing here. I believe it's best to first file it with PyWin and find out what it is trying to do and why.

Pyclbr behavior sort of makes sense to me -- pyclbr goal is to parse and browse the source of a module, but the __main__ module is either the current script, in which case it can be imported under another name (and then it would have the __spec__), or it's an interactive session, which you probably don't need to browse (maybe?).

I am from PythonWin. It's not a problem of PythonWin. That (an IDE / Editor / Debugger app) just uses pyclbr in a basic way to inspect any (user) module: data = pyclbr.readmodule_ex(mod)

It seems to me that PyWin is calling pyclbr readmodule_ex with module='__main__'

No, there's no explicit inspection of __main__. Contrarily, it's necessary to avoid the inspection - when indirectly triggered within pyclbr by an "import" statement in any module.

As mentioned, the problem happens when any module inspected directly or indirectly (via nested imports) by pyclbr does for example import __main__ or another import of any special / custom module with no real .__spec__
The __spec__ is optional info. But pyclr crashes when not present.

Easy to reproduce, just add an import __main__ to any test module file.
E.g. a case in PythonWin itself, pywin.framework.scriptutils:

import sys
import win32ui
import win32api
import win32con
import __main__
from pywin.mfc import dialog
...

It's a frequent case. Many apps / scripts get the startup script namespace as __main__.

Of course, one can monkey patch / workaround like __main__.__spec__ = ModuleSpec('__main__', None) in a certain known use case. But it's not anywhere a general solution.
pyclbr should just not crash, but behave like _ModuleBrowser.visit_Import() comment itself says:

        for module in node.names:
            ...
                # If we can't find or parse the imported module,
                # too bad -- don't die here.
                continue

I patch it like this so far:

--- a/pyclbr.py
+++ b/pyclbr.py
@@ -158,12 +158,16 @@ def _readmodule(module, path, inpackage=None):
         return _readmodule(submodule, parent['__path__'], package)
 
     # Search the path for the module.
-    f = None
     if inpackage is not None:
         search_path = path
     else:
         search_path = path + sys.path
-    spec = importlib.util._find_spec_from_path(fullmodule, search_path)
+    try:
+        spec = importlib.util._find_spec_from_path(fullmodule, search_path)
+    except ValueError as ev:
+        if '__spec__ is None' not in str(ev):  # nitpicky?
+            raise
+        spec = None
     if spec is None:
         raise ModuleNotFoundError(f"no module named {fullmodule!r}", name=fullmodule)
     _modules[fullmodule] = tree

@ncoghlan

If I create a directory with __main__.py and test.py with test.py containing import __main__ and class A: pass and __main__.py containing class B: pass, the __main__ being inspected is the one from pyclbr and not my own main:

.
├── __main__.py
└── test.py

This happens because I'm using -m option.

Now, if I were to invoke pyclbr as a regular script and not a module, I indeed get the same error, namely:

$ python Lib/pyclbr.py test.py
ModuleSpec(name='test', loader=<_frozen_importlib_external.SourceFileLoader object at 0x7f5b38ead6e0>, origin='~/projects/python/scratch/test.py')
ValueError: __main__.__spec__ is None

Unfortunately, the cached sys.modules is still the pyclbr main module. IOW, import __main__ is actually not importing the expected __main__ because when _main() of pyclbr is executed, the __main__ module is already cached. So we should first patch _main() by popping __main__ so that we don't import the wrong one.

Now, programatically:

# chk.py

import os
import pyclbr

d = os.path.expandvars("$HOME/projects/python/scratch")
print(pyclbr.readmodule_ex("test", [d]))

# ~/projects/python/scratch/test/test.py
import __main__
class A: pass

# ~/projects/python/scratch/test/__main__.py
class B: pass

and by adding sys.modules.pop('__main__', None) at the top of pyclbr.py file, the introspection works correctly (the correct modules are picked up):

$ python chk.py
ModuleSpec(name='test', loader=<_frozen_importlib_external.SourceFileLoader object at 0x7ff685a26bb0>, origin='$HOME/projects/python/scratch/test.py')
ModuleSpec(name='__main__', loader=<_frozen_importlib_external.SourceFileLoader object at 0x7ff685940a70>, origin='$HOME/projects/python/scratch/__main__.py')

Without sys.modules.pop('__main__', None), pyclbr would not work as it's trying to import pyclbr.py as the __main__ and if we don't do it programatically, it's still trying to use chk.py. Whatever choice it's making, the problem is that most of the time, the __main__ module that will be found would have None as its spec. So I suggest that we temporarily uncache __main__ so that import __main__ are treated as regular imports and not special ones.

I am not an expert in this area so I'm tagging some experts here @ncoghlan @brettcannon @ambv (since it looks like something magical that the REPL may also have for some reasons).

Given pyclbr has its own module cache, it should probably just be calling importlib._bootstrap._find_spec directly, and bypassing the sys.modules cache entirely (the sys.modules check is the key thing that importlib._util._find_spec_from_path adds over the lower level API, and it isn't a behaviour that pyclbr actually wants).

pythongh-87465: Fix pyclbr.readmodule_ex crash on modules with __spec…

Navigation Menu

Repository navigation

pyclbr.readmodule_ex traversing "import main": dies with ValueError: main.spec is None / is not set #87465

Linked PRs

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

pyclbr.readmodule_ex traversing "import __main__": dies with ValueError: __main__.__spec__ is None / is not set #87465

Description

Linked PRs

Activity

kxrob commented on Feb 22, 2021

akulakov commented on Jul 6, 2021

kxrob commented on Jul 10, 2021

akulakov commented on Jul 12, 2021

akulakov commented on Jul 12, 2021

akulakov commented on Jul 12, 2021

akulakov commented on Jul 12, 2021

kxrob commented on Aug 22, 2025

picnixz commented on Aug 28, 2025

ncoghlan commented on Aug 28, 2025

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions

pyclbr.readmodule_ex traversing "import main": dies with ValueError: main.spec is None / is not set #87465