Skip to content

Replace Python parser with ruff parser #5494

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
coolreader18 merged 75 commits into from
Mar 20, 2025
Merged

Replace Python parser with ruff parser #5494

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
75 commits
Select commit Hold shift + click to select a range
93b7d6b
stage1
qingshi163 Sep 10, 2024
e0a3f26
compiler pass build
qingshi163 Oct 7, 2024
38575bd
introduce rustpython-compiler-source
qingshi163 Oct 7, 2024
dce0d94
stage2
qingshi163 Oct 15, 2024
3162b9b
fixup
qingshi163 Oct 15, 2024
1f7024a
pass compile
qingshi163 Oct 28, 2024
e189ecf
Fix hello world compiler test
etaloof Dec 2, 2024
0f100e9
Fix code generation for if-elif-else statement
etaloof Dec 2, 2024
b42f88f
Fix code generation for lambda expression
etaloof Dec 2, 2024
7c53857
Fix code generation for integers
etaloof Dec 2, 2024
424e05a
Fix code generation for fstrings
etaloof Dec 2, 2024
2691cbe
Fix code generation for if statement
etaloof Dec 2, 2024
a98a623
Fix code generation for if statement
etaloof Dec 2, 2024
168aa01
Fix code generation for if statement
etaloof Dec 2, 2024
c25d958
Fix code generation for fstring
etaloof Dec 2, 2024
4e29dd4
Fix code generation for class definition
etaloof Dec 2, 2024
89afad2
Replace feature flags
etaloof Dec 2, 2024
e760ac7
Initialize frozen core modules
etaloof Dec 2, 2024
2987cbe
Allow __future__ import after module doc comment
etaloof Dec 2, 2024
c10df16
Disable ast module
etaloof Dec 2, 2024
bcfbf88
Commit remaining fixes for compile errors in examples
etaloof Dec 2, 2024
def0e42
Fix some warnings
etaloof Dec 6, 2024
4bef13b
Update ast stdlib module
etaloof Jan 3, 2025
1fd48c7
Update ast stdlib module
etaloof Jan 3, 2025
ca2b302
Update ast stdlib module
etaloof Jan 4, 2025
e9bc377
Update ast stdlib module
etaloof Jan 6, 2025
c6ece03
Update ast stdlib module
etaloof Jan 6, 2025
5c22c39
Split ast stdlib module into files
etaloof Jan 6, 2025
6b49cfb
Fix codegen for positional arguments with defaults
etaloof Jan 6, 2025
4cdd4ae
Update ast stdlib module
etaloof Jan 7, 2025
f03d43f
Update ast stdlib module
etaloof Jan 8, 2025
60dc970
Extract string and constant handling from expression.rs
etaloof Jan 8, 2025
5bd4d5b
Always add required fields to AST nodes
etaloof Jan 12, 2025
85edb25
Compile doc strings correctly again
etaloof Jan 12, 2025
f4154a0
Enable "ast" Cargo feature by default
etaloof Jan 12, 2025
1bff3dc
Refactor compilation of big integer literal
etaloof Jan 12, 2025
a669803
Update ast stdlib module
etaloof Jan 13, 2025
1bd4078
Update ast stdlib module
etaloof Jan 18, 2025
a054ca6
Update ast stdlib module
etaloof Jan 23, 2025
1476b49
Reset barebones example
etaloof Jan 23, 2025
1a3c5bb
Merge remote-tracking branch 'refs/remotes/origin/main' into ruff-parser
etaloof Jan 25, 2025
595622b
Fix some left-over warnings
etaloof Jan 25, 2025
c00d58f
Undo accidential change
etaloof Jan 25, 2025
d08716f
Adapt shell to ruff parser
etaloof Jan 25, 2025
c8acbf8
Pin parser to v0.4.10
youknowone Jan 27, 2025
c04f702
fix clippy
youknowone Jan 27, 2025
20dcf05
Add TODO about interactive mode
etaloof Jan 27, 2025
658b125
Fix compilation of complex number expression
etaloof Jan 28, 2025
ce9eeaf
Remove moved code
etaloof Jan 28, 2025
2f8ca1f
Update test case to ruff v0.4.10
etaloof Jan 28, 2025
3c078bc
Apply suggestion
etaloof Jan 29, 2025
591757b
Apply suggestion
etaloof Jan 29, 2025
494f626
Fix compilation of fstring expression
etaloof Feb 3, 2025
36315e3
Fix compilation of fstring expression
etaloof Feb 4, 2025
afdf27a
Fix wasm compile errors
etaloof Feb 6, 2025
de278fc
Attach correct source locations to ast objects
etaloof Feb 7, 2025
624cdb2
Fix some more wasm compile errors
etaloof Feb 10, 2025
68ab9aa
Consider compile mode and enable AST stdlib module again
etaloof Feb 17, 2025
3fce4be
Fix incorrect AST source location end column
etaloof Feb 17, 2025
9b14ad6
Fix compile error if "compiler" feature is not enabled
etaloof Feb 22, 2025
47a4862
Merge branch 'main' into ruff-parser
coolreader18 Feb 25, 2025
f770155
Fix regrtests
coolreader18 Feb 25, 2025
79b627e
Fix some test_ast tests
coolreader18 Feb 25, 2025
7128b8c
Merge remote-tracking branch 'origin/main' into ruff-parser
etaloof Feb 25, 2025
5cc508d
Add source range to type ignore
etaloof Mar 15, 2025
86668ee
Fix incompatibility with Rust 2024 edition
etaloof Mar 15, 2025
16ac422
Fix todos by implementing missing ast conversions and deleting unused…
etaloof Mar 15, 2025
77d49d2
Merge branch 'main' into ruff-parser
coolreader18 Mar 19, 2025
7c888d9
Merge branch 'main' into ruff-parser
coolreader18 Mar 19, 2025
b90d02e
Appease clippy
coolreader18 Mar 19, 2025
e328031
Fix remaining ast tests
coolreader18 Mar 19, 2025
23cc9b8
Fix remaining ast tests
coolreader18 Mar 19, 2025
226b677
Mark/fix remaining tests
coolreader18 Mar 19, 2025
c15da44
Fix more
coolreader18 Mar 19, 2025
85e40b3
Hacky windows fix
coolreader18 Mar 20, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
216 changes: 125 additions & 91 deletions Cargo.lock
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

16 changes: 11 additions & 5 deletions Cargo.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ rustpython-compiler = { workspace = true }
rustpython-pylib = { workspace = true, optional = true }
rustpython-stdlib = { workspace = true, optional = true, features = ["compiler"] }
rustpython-vm = { workspace = true, features = ["compiler"] }
rustpython-parser = { workspace = true }
ruff_python_parser = { workspace = true }

cfg-if = { workspace = true }
log = { workspace = true }
Expand Down Expand Up @@ -94,7 +94,7 @@ x86_64-pc-windows-msvc = { triplet = "x64-windows-static-md", dev-dependencies =
[workspace]
resolver = "2"
members = [
"compiler", "compiler/core", "compiler/codegen",
"compiler", "compiler/core", "compiler/codegen", "compiler/source",
".", "common", "derive", "jit", "vm", "vm/sre_engine", "pylib", "stdlib", "derive-impl",
"wasm/lib",
]
Expand All @@ -108,6 +108,7 @@ repository = "https://github.com/RustPython/RustPython"
license = "MIT"

[workspace.dependencies]
rustpython-compiler-source = { path = "compiler/source" }
rustpython-compiler-core = { path = "compiler/core", version = "0.4.0" }
rustpython-compiler = { path = "compiler", version = "0.4.0" }
rustpython-codegen = { path = "compiler/codegen", version = "0.4.0" }
Expand All @@ -121,15 +122,20 @@ rustpython-stdlib = { path = "stdlib", default-features = false, version = "0.4.
rustpython-sre_engine = { path = "vm/sre_engine", version = "0.4.0" }
rustpython-doc = { git = "https://github.com/RustPython/__doc__", tag = "0.3.0", version = "0.3.0" }

ruff_python_parser = { git = "https://github.com/astral-sh/ruff.git", tag = "v0.4.10" }
ruff_python_ast = { git = "https://github.com/astral-sh/ruff.git", tag = "v0.4.10" }
ruff_text_size = { git = "https://github.com/astral-sh/ruff.git", tag = "v0.4.10" }
ruff_source_file = { git = "https://github.com/astral-sh/ruff.git", tag = "v0.4.10" }
ruff_python_codegen = { git = "https://github.com/astral-sh/ruff.git", tag = "v0.4.10" }
# rustpython-literal = { version = "0.4.0" }
# rustpython-parser-core = { version = "0.4.0" }
# rustpython-parser = { version = "0.4.0" }
# rustpython-ast = { version = "0.4.0" }
# rustpython-format= { version = "0.4.0" }
rustpython-literal = { git = "https://github.com/RustPython/Parser.git", version = "0.4.0", rev = "d2f137b372ec08ce4a243564a80f8f9153c45a23" }
rustpython-parser-core = { git = "https://github.com/RustPython/Parser.git", version = "0.4.0", rev = "d2f137b372ec08ce4a243564a80f8f9153c45a23" }
rustpython-parser = { git = "https://github.com/RustPython/Parser.git", version = "0.4.0", rev = "d2f137b372ec08ce4a243564a80f8f9153c45a23" }
rustpython-ast = { git = "https://github.com/RustPython/Parser.git", version = "0.4.0", rev = "d2f137b372ec08ce4a243564a80f8f9153c45a23" }
# rustpython-parser-core = { git = "https://github.com/RustPython/Parser.git", version = "0.4.0", rev = "d2f137b372ec08ce4a243564a80f8f9153c45a23" }
# rustpython-parser = { git = "https://github.com/RustPython/Parser.git", version = "0.4.0", rev = "d2f137b372ec08ce4a243564a80f8f9153c45a23" }
# rustpython-ast = { git = "https://github.com/RustPython/Parser.git", version = "0.4.0", rev = "d2f137b372ec08ce4a243564a80f8f9153c45a23" }
rustpython-format = { git = "https://github.com/RustPython/Parser.git", version = "0.4.0", rev = "d2f137b372ec08ce4a243564a80f8f9153c45a23" }
# rustpython-literal = { path = "../RustPython-parser/literal" }
# rustpython-parser-core = { path = "../RustPython-parser/core" }
Expand Down
Loading