option to speed-up local check build - #2071
Merged
Merged
Conversation
Profiling the real build logs showed Step 3 is 62-90% of a slim build,
and inside it resolve is 1.5% while install is 96.8%. The built slim
distro carries 38,590 .pyc for 38,590 .py: pip byte-compiles every
module, serially, and that is most of the 13.6 minutes.
Most local builds only exist to produce a pylock or pre-check the final
size, and do not need .pyc at all. So make it a choice, per build:
bytecode = "pip" # default: inline, exactly what ships today
bytecode = "none" # no .pyc -- throwaway builds
bytecode = "parallel" # --no-compile, then compileall over every core
bytecode = "parallel-4" # ... capped at 4 workers
Non-default modes pass --no-compile to pip and, for parallel, run one
compileall pass with the *target* interpreter after Step 4, so the
patched sources are the ones compiled. Compile failures stay non-fatal,
matching pip: some packages ship modules that will not compile here.
Measured on the build machine (8 cores, Defender realtime on), 2520-file
sample: compileall -j1 45s cold / 36s warm, -j0 18.7s. About 2x rather
than 8x, because the work is partly I/O-scan-bound rather than CPU-bound
-- worth having, but not the whole story. Antivirus exclusions for the
build trees are likely a bigger lever and are the user's call.
Default is unchanged, so release builds keep shipping .pyc exactly as
before. A typo in the TOML raises rather than silently shipping none.
tests/test_build_options.py covers the mode parsing, and winpython/** is
added to the CI path filter so build changes now run the suite. Wiring
verified end to end: existing TOMLs still yield --bytecode pip, a TOML
setting it yields it for that flavor only, the value survives
build_winpython's parser, and the emitted compileall command targets the
right interpreter and site-packages.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Moving byte-compilation out of pip surfaced ~11 SyntaxWarnings per slim
build that nobody had seen before: invalid escape sequences in pydub and
ipywidgets, "'return' in a 'finally' block" in winappdbg, mssql_python,
sympy and win32comext.
They are not new, and nothing is wrong with the packages' .pyc. pip
wraps its own byte-compilation in warnings.filterwarnings("ignore")
(pip/_internal/operations/install/wheel.py), so it never showed them.
compileall as a CLI does not, so pass -W ignore and match pip.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
os.cpu_count() reports logical CPUs, so on the build laptop (i7-8550U, 4 cores / 8 threads) an uncapped default started 8 workers. Measured, the compile is I/O- and antivirus-bound rather than CPU-bound -- -j4 beat pip's serial inline compile by only 11% on a real slim build -- so the extra 4 SMT workers buy contention, not speed. "parallel" now means min(os.cpu_count(), 4). An explicit "parallel-N" stays literal and uncapped, since that is a deliberate choice. Also rejects parallel-0, which would have handed the count back to ProcessPoolExecutor and undone the cap. Deliberately os.cpu_count() and not os.process_cpu_count(): builds run on Python 3.10 (WPy64-310111) where the latter does not exist, while CI tests 3.13/3.14 and would not have caught it. Verified directly on 3.10.11. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.