Skip to content

option to speed-up local check build - #2071

Merged
stonebig merged 3 commits into
winpython:masterfrom
stonebig:master
Aug 9, 2026
Merged

option to speed-up local check build#2071
stonebig merged 3 commits into
winpython:masterfrom
stonebig:master

Conversation

@stonebig

@stonebig stonebig commented Aug 9, 2026

Copy link
Copy Markdown
Contributor

No description provided.

stonebig and others added 3 commits August 9, 2026 12:04
Profiling the real build logs showed Step 3 is 62-90% of a slim build,
and inside it resolve is 1.5% while install is 96.8%. The built slim
distro carries 38,590 .pyc for 38,590 .py: pip byte-compiles every
module, serially, and that is most of the 13.6 minutes.

Most local builds only exist to produce a pylock or pre-check the final
size, and do not need .pyc at all. So make it a choice, per build:

    bytecode = "pip"         # default: inline, exactly what ships today
    bytecode = "none"        # no .pyc -- throwaway builds
    bytecode = "parallel"    # --no-compile, then compileall over every core
    bytecode = "parallel-4"  # ... capped at 4 workers

Non-default modes pass --no-compile to pip and, for parallel, run one
compileall pass with the *target* interpreter after Step 4, so the
patched sources are the ones compiled. Compile failures stay non-fatal,
matching pip: some packages ship modules that will not compile here.

Measured on the build machine (8 cores, Defender realtime on), 2520-file
sample: compileall -j1 45s cold / 36s warm, -j0 18.7s. About 2x rather
than 8x, because the work is partly I/O-scan-bound rather than CPU-bound
-- worth having, but not the whole story. Antivirus exclusions for the
build trees are likely a bigger lever and are the user's call.

Default is unchanged, so release builds keep shipping .pyc exactly as
before. A typo in the TOML raises rather than silently shipping none.

tests/test_build_options.py covers the mode parsing, and winpython/** is
added to the CI path filter so build changes now run the suite. Wiring
verified end to end: existing TOMLs still yield --bytecode pip, a TOML
setting it yields it for that flavor only, the value survives
build_winpython's parser, and the emitted compileall command targets the
right interpreter and site-packages.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Moving byte-compilation out of pip surfaced ~11 SyntaxWarnings per slim
build that nobody had seen before: invalid escape sequences in pydub and
ipywidgets, "'return' in a 'finally' block" in winappdbg, mssql_python,
sympy and win32comext.

They are not new, and nothing is wrong with the packages' .pyc. pip
wraps its own byte-compilation in warnings.filterwarnings("ignore")
(pip/_internal/operations/install/wheel.py), so it never showed them.
compileall as a CLI does not, so pass -W ignore and match pip.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
os.cpu_count() reports logical CPUs, so on the build laptop (i7-8550U,
4 cores / 8 threads) an uncapped default started 8 workers. Measured, the
compile is I/O- and antivirus-bound rather than CPU-bound -- -j4 beat
pip's serial inline compile by only 11% on a real slim build -- so the
extra 4 SMT workers buy contention, not speed.

"parallel" now means min(os.cpu_count(), 4). An explicit "parallel-N"
stays literal and uncapped, since that is a deliberate choice. Also
rejects parallel-0, which would have handed the count back to
ProcessPoolExecutor and undone the cap.

Deliberately os.cpu_count() and not os.process_cpu_count(): builds run on
Python 3.10 (WPy64-310111) where the latter does not exist, while CI
tests 3.13/3.14 and would not have caught it. Verified directly on
3.10.11.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@stonebig
stonebig merged commit 5e154be into winpython:master Aug 9, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant