Btw, while testing this PR I found that some of the benchmark scripts are broken (for example make time_stdlib_parse):

The venv has been created in the ./venv directory
./venv/bin/python scripts/benchmark.py --parser=cpython --target=xxl parse
Traceback (most recent call last):
  File "/home/pablogsal/github/python/master/Tools/peg_generator/scripts/benchmark.py", line 17, in <module>
    from peg_extension import parse
ImportError: cannot import name 'parse' from 'peg_extension' (/home/pablogsal/github/python/master/Tools/peg_generator/peg_extension/__init__.py)
make: *** [Makefile:87: time_stdlib_parse] Error 1

given that we are focusing on the Python parser now, maybe we could adapt those to use the _peg_parser extension instead of trying to compile the parser.

Btw, while testing this PR I found that some of the benchmark scripts are broken (for example make time_stdlib_parse):

The venv has been created in the ./venv directory
./venv/bin/python scripts/benchmark.py --parser=cpython --target=xxl parse
Traceback (most recent call last):
  File "/home/pablogsal/github/python/master/Tools/peg_generator/scripts/benchmark.py", line 17, in <module>
    from peg_extension import parse
ImportError: cannot import name 'parse' from 'peg_extension' (/home/pablogsal/github/python/master/Tools/peg_generator/peg_extension/__init__.py)
make: *** [Makefile:87: time_stdlib_parse] Error 1

given that we are focusing on the Python parser now, maybe we could adapt those to use the _peg_parser extension instead of trying to compile the parser.

The problem is that _peg_parser in its current state only parses and generates the external AST. It cannot compile to byte code and it cannot just parse the internal AST. Should we just add build to the dependencies of make time_stdlib_parse?

The problem is that _peg_parser in its current state only parses and generates the external AST.

I see. In any case I suppose that comparing with the old parser will be something that we don't want to do anymore (and the old parser will be removed for the 3.10 series) so I wonder how many of these scripts should be deleted or just ported to use ast.parse or compile directly. For instance, the scripts/test_pypi_packages.py has some logic to compile the parser that is not needed now that is the default one.

Could we do a review of the scripts we want to keep around and delete the rest?

Could we do a review of the scripts we want to keep around and delete the rest?

Added to my to-do list!

Should we merge this as-is?

Should we merge this as-is?

Yeah, I think is still an improvement 👌

@lysnikolaou: Status check is done, and it's a success ✅ .