Repository navigation
Closed
Labels
3.12only security fixesonly security fixes3.13only security fixesonly security fixesstdlibStandard Library Python modules in the Lib/ directoryStandard Library Python modules in the Lib/ directorytype-bugAn unexpected behavior, bug, or errorAn unexpected behavior, bug, or error
Description
Bug report
Bug description:
The getpath module tries to read certain _pth files during initialization).
This is tested in test_site with generated _pth files that include the stdlib path 200 times.
The getpath module disallows reading files over 32KB during initialization.
If the test suite runs from a very long base path, 200 repetitions of the stdlib path in the _pth file would be enough to exceed the 32KB limit.
To demonstrate, artificially increase the number of repetitions to some high number that would exceed 32KB, e.g. this patch:
diff --git a/Lib/test/test_site.py b/Lib/test/test_site.py
index 33d0975bda8..ba2f7d90a80 100644
--- a/Lib/test/test_site.py
+++ b/Lib/test/test_site.py
@@ -668,7 +668,7 @@ def test_underpth_nosite_file(self):
exe_prefix = os.path.dirname(sys.executable)
pth_lines = [
'fake-path-name',
- *[libpath for _ in range(200)],
+ *[libpath for _ in range(5000)],
'',
'# comment',
]and observe the test failing with the following error:
$ ./python.exe -m test test_site -v -m '*_pthFileTests.test_underpth_nosite_file'
== CPython 3.13.0a2+ (heads/main-dirty:b4b2cc10121, Jan 1 2024, 11:38:27) [Clang 15.0.0 (clang-1500.1.0.2.5)]
== macOS-14.2.1-arm64-arm-64bit little-endian
== Python build: debug
== cwd: /Users/itamaro/work/pyexe/main-dbg/build/test_python_worker_96404æ
== CPU count: 12
== encodings: locale=UTF-8 FS=utf-8
== resources: all test resources are disabled, use -u option to unskip tests
Using random seed: 4200167786
0:00:00 load avg: 8.99 Run 1 test sequentially
0:00:00 load avg: 8.99 [1/1] test_site
test_underpth_nosite_file (test.test_site._pthFileTests.test_underpth_nosite_file) ... Exception ignored in running getpath:
Traceback (most recent call last):
File "<frozen getpath>", line 462, in <module>
MemoryError: cannot read file larger than 32KB during initialization
Fatal Python error: error evaluating path
Python runtime state: core initialized
Current thread 0x00000001e0e39000 (most recent call first):
<no Python frame>
ERROR
======================================================================
ERROR: test_underpth_nosite_file (test.test_site._pthFileTests.test_underpth_nosite_file)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/Users/itamaro/work/cpython/Lib/test/test_site.py", line 683, in test_underpth_nosite_file
output = subprocess.check_output([exe_file, '-c',
~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
'import sys; print("\\n".join(sys.path) if sys.flags.no_site else "")'
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
], env=env, encoding='utf-8', errors='surrogateescape')
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/itamaro/work/cpython/Lib/subprocess.py", line 470, in check_output
return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
**kwargs).stdout
^^^^^^^^^
File "/Users/itamaro/work/cpython/Lib/subprocess.py", line 575, in run
raise CalledProcessError(retcode, process.args,
output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['/var/folders/ng/hmf622ys3wnc3c21prtx7nd80000gn/T/tmpkc8z54mu/python.exe', '-c', 'import sys; print("\\n".join(sys.path) if sys.flags.no_site else "")']' returned non-zero exit status 1.
----------------------------------------------------------------------
Ran 1 test in 0.032s
FAILED (errors=1)
test test_site failed
test_site failed (1 error)
== Tests result: FAILURE ==
1 test failed:
test_site
Total duration: 150 ms
Total tests: run=1 (filtered)
Total test files: run=1/1 (filtered) failed=1
Result: FAILURECPython versions tested on:
3.12, 3.13, CPython main branch
Operating systems tested on:
Linux, macOS
Linked PRs
Metadata
Metadata
Assignees
Labels
3.12only security fixesonly security fixes3.13only security fixesonly security fixesstdlibStandard Library Python modules in the Lib/ directoryStandard Library Python modules in the Lib/ directorytype-bugAn unexpected behavior, bug, or errorAn unexpected behavior, bug, or error
Activity
itamaro commentedon Jan 1, 2024
A few ways to go about this:
test_site(e.g. from 200 to 20) to raise the threshold for the path length that would trigger this failure (this fixes it for me internally). I don't know what's the significance of the number of repetitions in these tests, and why 200 was chosen specifically._pthfile exceeds 32KB, the test skips itself (e.g. raise some resource denied)._pthgeneration "adaptive" - start with 200, but as long as the file exceeds 32KB try fewer repetitions (e.g. 100, 50, 20), and skip the test if even the fewest number of repetitions exceeds 32KB.I'm happy to submit a PR for any of the 2-4 options.
terryjreedy commentedon Jan 1, 2024
A test that fails with a legal stdlib path over about 160 (2**15//200 rounded down) chars is broken. I would replace 200 with
min(200, 30000 // len(libpath)).fix pythongh-113628: reduce number of libpath lines in pth lines to a…
gh-113628: Fix test_site test with long stdlib paths (#113640)
pythongh-113628: Fix test_site test with long stdlib paths (pythonGH-…
pythongh-113628: Fix test_site test with long stdlib paths (pythonGH-…
[3.12] gh-113628: Fix test_site test with long stdlib paths (GH-113640)…
[3.11] gh-113628: Fix test_site test with long stdlib paths (GH-113640)…
Backport gh-113628: Fix test_site test with long stdlib paths
pythongh-113628: Fix test_site test with long stdlib paths (python#11…
pythongh-113628: Fix test_site test with long stdlib paths (python#11…
pythongh-113628: Fix test_site test with long stdlib paths (python#11…