Skip to content

Speedup compile time for rustpython-parser - #3652

Merged
coolreader18 merged 2 commits into
RustPython:mainfrom
coolreader18:speedup-parser-compilation
Apr 27, 2022
Merged

coolreader18 merged 2 commits into
RustPython:mainfrom
coolreader18:speedup-parser-compilation

Conversation

@coolreader18

Copy link
Copy Markdown
Member

Saves about half a minute on a clean build, see timing reports:

Before

Targets: rustpython 0.1.2 (lib, bin "rustpython")
Profile: release
Fresh units: 0
Dirty units: 345
Total units: 345
Max concurrency: 8 (jobs=8 ncpu=8)
Build start: 2022-04-22T19:33:33Z
Total time: 177.4s (2m 57.4s)

After

Targets: rustpython 0.1.2 (lib, bin "rustpython")
Profile: release
Fresh units: 0
Dirty units: 320
Total units: 320
Max concurrency: 8 (jobs=8 ncpu=8)
Build start: 2022-04-22T22:09:37Z
Total time: 153.6s (2m 33.6s)

@coolreader18
coolreader18 force-pushed the speedup-parser-compilation branch from ec63ea3 to 9649ea5 Compare April 22, 2022 22:25
Comment thread parser/build.rs

fn check_lalrpop() {
println!("cargo:rerun-if-changed=src/python.lalrpop");
let sha3_line = BufReader::with_capacity(128, File::open("src/python.rs").unwrap())

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we are going to generate this file, I think the output is expected not to be placed in repository but loaded form CI cache.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it's fine to check it into the repo; while different cause it's a bootstrapping issue, lalrpop itself checks its generated parser into its repo, I feel like it's a decently common practice. While I do really like the idea of having everything necessary for the build be generated at build-time, that comes in conflict with the reality of longer build times. If it verifies that the generated file is up-to-date, I don't think I really see an issue with it.

@youknowone youknowone Apr 22, 2022 •

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think that's not because of lalrpop convention but a typical compiler bootstrapping issue. If they don't bundle generated parser to their repository, people who don't have lalrpop yet will never can generate lalrpop parser by themselves.

I am not sure if I am little bit out of trends, but as I understand, traditional software communities only commit the original source to the repository but attaching the configuration result to their source code distribution. We usually now don't distribute source code anymore. So we'd better to invest more on easier build from source, but still we can avoid to bundle the generated output itself when possible.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

practically, once after building the parser first, it will not take long time again because contributors and users never edit parsers. So this is concerning about CI, not the users or contributors, right?
then I think we can build this from CI and cache the result. There will be another dependency, but it will not be a big deal after first build because cache will be shared between the different CI runs.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Another advantage to including the parser in the repository is that we can see how much of it is covered by our test suite.

@youknowone youknowone Apr 25, 2022 •

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That point makes sense. We need rustfmt in that case.

@coolreader18

Copy link
Copy Markdown
Member Author

Achh, newline issues on windows.

@fanninpm

fanninpm commented Apr 23, 2022 •

Copy link
Copy Markdown
Contributor

TAKE MY BENCHMARKS WITH A GRAIN OF SALT

HP Elitebook 745 G3
Manjaro Stable (2022-04-15 update)
Rust v1.60.0
*processes are running in the background*
listening to some Bach on another computer

Current main (c103fe1)

Rendered

Targets: rustpython 0.1.2 (lib, bin "rustpython")
Profile: release
Fresh units: 0
Dirty units: 375
Total units: 375
Max concurrency: 4 (jobs=4 ncpu=4)
Build start: 2022-04-23T01:04:52Z
Total time: 726.9s (12m 6.9s)
rustc: rustc 1.60.0 (7737e0b5c 2022-04-04)
Host: x86_64-unknown-linux-gnu
Target: x86_64-unkown-linux-gnu
Max (global) rustc threads concurrency: 0

This branch (2918395)

Rendered

Targets: rustpython 0.1.2 (lib, bin "rustpython")
Profile: release
Fresh units: 0
Dirty units: 357
Total units: 357
Max concurrency: 4 (jobs=4 ncpu=4)
Build start: 2022-04-23T01:38:57Z
Total time: 575.9s (9m 35.9s)
rustc: rustc 1.60.0 (7737e0b5c 2022-04-04)
Host: x86_64-unknown-linux-gnu
Target: x86_64-unkown-linux-gnu
Max (global) rustc threads concurrency: 0

@coolreader18
coolreader18 force-pushed the speedup-parser-compilation branch from 2918395 to ad3dcb9 Compare April 24, 2022 15:36
@coolreader18

Copy link
Copy Markdown
Member Author

Should I rustfmt it?

@coolreader18
coolreader18 merged commit 996fe65 into RustPython:main Apr 27, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants