pattern matching parser support - #4323

This would be huge for Ruff.

ah, its parser part was not that much ready and I need to find how to add a soft-keyword concept to parser.

I hope this can help Ruff to support pattern matching and close the issue astral-sh/ruff#282

The dream...

Sorry to double-post, but copying over my most recent comment from the related issue in Ruff:

I'd love to sponsor someone (financially, that is) to work on this issue (in RustPython). If you're interested, shoot me a DM on Twitter or email me at charlie.r.marsh@gmail.com :)

To be honest, after watching @andersk's work, I thought his way would be much better than my hacky local work.
I was going to add a mode to lexer to optionally generate a match token - which is not a good idea for either design or performance.
@andersk Do you have a plan to work on it? I think you are the one who is most close to the goal.

Another possibility is that if @andersk has any guidance to share on how he thought this might be implemented (per prior comments), I could give it a try. I've put off going deep on LALRPOP :)

I’m still slowly working on this. I have a rough plan for splitting the grammar that ought to work with LR(1) theoretically, but I took a detour to investigate what seem to be LALRPOP bugs. One fix I wrote is lalrpop/lalrpop#697 (I wonder if LALRPOP still has maintainers who feel empowered to merge it…), but there are other issues I haven’t finished diagnosing.

I’m not entirely sure at this point whether I’m going down this path because I think it will result in the best parser or just because it seems like an interesting challenge. Switching to a more powerful parser algorithm may be the best long-term plan, especially if Python continues adding more tricky constructs to the grammar.

Switching to a more powerful parser algorithm may be the best long-term plan, especially if Python continues adding more tricky constructs to the grammar.

“May be” might be an understatement. A parser library that can fundamentally handle the complexity that Python’s grammar has is definitely better than using one that doesn’t and hacking around its limitations, right? Guido wouldn’t have replaced Python’s own parser unless that was the case. (The PEP mentions these hacks in its introduction paragraph)

Totally agree that we should look to upgrade the parser if we believe LALRPOP will be insufficient in the long-term. From a pragmatic perspective, though, I still think there's a lot of value in supporting structural pattern matching via the current parser if possible, even if it requires some hacks.

I wasn’t suggesting that LALRPOP wasn’t enough! Python’s old parser was LL(1), i.e. fixed-length lookahead only. I think that’s where the hacks came in. LALRPOP more powerful than that and I think that that’s probably sufficiently powerful to parse the whole grammar without hacks.