Repository navigation
Speed up the parser #153568
Description
This is a catch-all issue to track a series of small, independent speedups to the parser
Linked PRs
- gh-153568: Skip parser memo lookups that cannot match #153571
- gh-153568: Make the parser's per-rule stack check an inline comparison #153573
- gh-153568: Stop re-walking memo lists when growing left-recursive rules #153574
- gh-153568: Skip the expression precedence chain for single-token atoms #153575
- gh-153568: Don't materialize parser token text that is never read #153576
- gh-153568: Cache repeated identifiers while parsing #153577
- gh-153568: Pause the GC while converting the AST to Python objects #153583
- gh-153568: Skip newline translation for source without carriage returns #153584
- gh-153568: Use FIRST sets to speed up parser choices #156209
- gh-153568: Reduce parser allocations #157374
- gh-153568: Parse arithmetic with operator loops #157413
- gh-153568: Factor shared prefixes in expression rules #157437
- gh-153568: Share expression parsing with assignment targets #157450
- gh-153568: Reuse shared prefixes in generated parser rules #157465
- gh-153568: Skip prefix tests when peg_generator is unavailable #157534
- gh-153568: Skip PEG alternatives that cannot match the next token #160458
- gh-153568: Use a managed dict for AST nodes #160616
Activity
pythongh-153568: Skip parser memo lookups that cannot match
StanFromIreland commentedon Jul 11, 2026
@pablogsal I have a little suggestion in #148931.
pablogsal commentedon Jul 11, 2026
Will take a look soon. Thanks a lot!
pythongh-153568: Skip the expression precedence chain for single-toke…
pythongh-153568: Skip the expression precedence chain for single-toke…
pythongh-153568: Skip the expression precedence chain for single-toke…
pythongh-153568: Skip the expression precedence chain for single-toke…
pythongh-153568: Skip the expression precedence chain for single-toke…
pythongh-153568: Add fast paths for single-token atoms in the parser
pythongh-153568: Add fast paths for single-token atoms in the parser
pythongh-153568: Skip the expression precedence chain for single-toke…
70 remaining items
gh-153568: Reuse shared prefixes in generated parser rules (#157465)
pythongh-153568: Use FIRST sets to speed up parser choices
pythongh-153568: Fix FIRST-set dispatch typing
pythongh-153568: Make FIRST-set dispatch conservative
pythongh-153568: Make FIRST-set dispatch conservative
gh-153568: Skip prefix tests when peg_generator is unavailable (#157534)
eendebakpt commentedon Sep 15, 2026
Improvements of the parser might help with dataclass creation. See https://discuss.python.org/t/faster-lazier-dataclasses/107744 for a discussion. Profiling showed a significant amount of time spend in the parser, so this might be a nice testcase.
For
performance increased by about 20% (tested on a53b5b1 vs 2fcb0e2 )
gh-153568: Reduce parser allocations (#157374)
pythongh-153568: Skip the expression precedence chain for single-toke…
pythongh-153568: Parse rule flags in the Rule constructor
pythongh-153568: Skip the expression precedence chain for single-toke…
pythongh-153568: Parse rule flags in the Rule constructor
gh-153568: Skip the expression precedence chain for single-token atoms (
gh-153568: Use a managed dict for AST nodes (#160616)