Skip to content

gh-153568: Reduce parser allocations - #157374

Open
pablogsal wants to merge 10 commits into
python:mainfrom
pablogsal:gh-153568-parser-allocations
Open

pablogsal wants to merge 10 commits into
python:mainfrom
pablogsal:gh-153568-parser-allocations

Conversation

@pablogsal

@pablogsal pablogsal commented Sep 12, 2026

Copy link
Copy Markdown
Member

Allocate tokens in blocks and keep text only for names, literals and type comments. Use a bit for the Barry spelling and put the keyword error messages in the grammar. Create the identifier cache on first use and reuse the newline buffer. Reuse parsed names and numbers during backtracking, and reset memo filters before the error pass.

Related to #153568.

Codebase Python files Parsing speed Peak parser memory
Standard library 2,060 8.0% faster 11.1% less
Home Assistant 18,731 4.4% faster 6.7% less
mypy 442 5.9% faster 10.7% less
PyTorch 4,890 7.2% faster 5.1% less

@pablogsal

Copy link
Copy Markdown
Member Author

This looks gigantic because the parser autogenerated changes but it's quite tiny :)

@pablogsal
pablogsal force-pushed the gh-153568-parser-allocations branch from ec20017 to e2e3e7d Compare September 14, 2026 20:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant