Skip to content

Speed up import isal.igzip by not importing the typing module - #319

Merged
rhpvorderman merged 1 commit into
pycompression:developfrom
ptweezy:lazy-typing-import
Nov 9, 2026
Merged

rhpvorderman merged 1 commit into
pycompression:developfrom
ptweezy:lazy-typing-import

Conversation

@ptweezy

@ptweezy ptweezy commented Nov 5, 2026 •

Copy link
Copy Markdown
Contributor

import isal.igzip and import isal.igzip_threaded also import the typing module, but the two modules use the names that they import from it only in annotations. typing in turn imports other modules, for example, functools on Python 3.14 and re, enum, and contextlib on Python 3.11, so every program that imports isal.igzip pays for them. python -m isal.igzip loads most of these modules anyway through argparse and shutil, so on the command line this PR saves only the import of typing itself.

With this PR, neither module imports typing:

  • igzip.py annotates the mtime parameter of compress() as int | None instead of Optional[SupportsInt]. Type checkers now report an mtime that isn't an int, but at run time compress() still converts mtime with int().
  • igzip_threaded.py uses list, tuple, and X | None in its annotations instead of List, Tuple, and Optional.

The test_import_does_not_load_cli_modules test is now test_import_does_not_load_slow_modules, and it also checks that typing isn't imported. That check fails on develop.

I measured the wall-clock time of new processes that run python -c "import isal.igzip", in paired rounds against develop. The Linux numbers are from Docker on WSL2, and the Windows numbers are from native builds. The table shows the medians in milliseconds:

Platform develop This PR
Linux, CPython 3.14 12.8 9.8
Linux, CPython 3.11 14.0 8.4
Linux, PyPy 3.11 30.9 21.0
Windows, CPython 3.14 40.5 34.0
Windows, CPython 3.11 40.9 27.4

On Linux with CPython 3.14, python -c pass takes 7.6 ms, so importing isal.igzip adds about 5.2 ms to the process on develop and 2.1 ms with this PR. On the five platforms in the table, python -m isal.igzip -c on a 64 KiB file finishes 1.3 to 2.4 ms sooner. A program that imports typing for other reasons saves nothing. xopen and dnaio both import typing, for example.

The test suite passes on this branch on Windows with CPython 3.14 and 3.11.

Checklist

  • Pull request details were added to CHANGELOG.rst
  • Documentation was updated (if needed)

@ptweezy

ptweezy commented Nov 5, 2026

Copy link
Copy Markdown
Contributor Author

This is very much a speedup that's tiny, but being me, I find each of them satisfying and I like how they add up when you consider users x runs

@rhpvorderman rhpvorderman left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me, one minor thing so it can be done with even less code.

Comment thread src/isal/igzip.py Resolved
Comment thread src/isal/igzip.py Resolved
@ptweezy
ptweezy force-pushed the lazy-typing-import branch from 3d815c7 to 7420b66 Compare November 6, 2026 15:53
Only type checkers need typing, and importing it also loads functools and
copyreg on Python 3.14, and re, enum, contextlib and warnings on 3.11.
igzip_threaded now uses built-in generics, and igzip annotates the mtime
argument of compress() as int | None instead of Optional[SupportsInt].
Under callgrind, import isal.igzip costs 10.5M instead of 28.4M
instructions on top of interpreter startup on Python 3.14, and 5.4M
instead of 38.6M on 3.11.
@ptweezy
ptweezy force-pushed the lazy-typing-import branch from 7420b66 to 4f3faf7 Compare November 6, 2026 15:55
@ptweezy
ptweezy requested a review from rhpvorderman November 6, 2026 15:57
@rhpvorderman

Copy link
Copy Markdown
Collaborator

As a bit of history context, when I started this library years ago the | was not yet available for types for all supported versions of python. These sort of maintenance updates are important.

Thank you for all your work, I have to focus on other projects now as well so it will take some time before I get to all the PRs.

@rhpvorderman
rhpvorderman merged commit 8bf9346 into pycompression:develop Nov 9, 2026
47 of 48 checks passed
@ptweezy
ptweezy deleted the lazy-typing-import branch November 9, 2026 12:48
@ptweezy

ptweezy commented Nov 9, 2026

Copy link
Copy Markdown
Contributor Author

As a bit of history context, when I started this library years ago the | was not yet available for types for all supported versions of python. These sort of maintenance updates are important.

Thank you for all your work, I have to focus on other projects now as well so it will take some time before I get to all the PRs.

No problem at all. Appreciate your reviews and feedback. Take all the time! The work never ends haha!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants