The main challenge here is that match and case are soft keywords. Naïvely adding them as keyword tokens will break parsing of code like

match = re.match(pattern, str)
if match is not None:
    print(match[1])

that uses match as an identifier.

We typically don’t know whether match should be treated as a keyword or an identifier until many tokens later. Some stupid examples I’ve collected where this results in interesting parse tree differences:

match *a + b, c   # ((match * a) + b), c
match *a + b, c:  # match ((*(a + b)), c):

match *(a + b), c   # (match * (a + b)), c
match *(a + b), c:  # match ((*(a + b)), c):

match (*a + b, c)   # match ((*(a + b)), c)
match (*a + b, c):  # match ((*(a + b)), c):

match -a * b + c   # (match - (a * b)) + c
match -a * b + c:  # match (((-a) * b) + c):

match -(a * b) + c   # (match - (a * b)) + c
match -(a * b) + c:  # match ((-(a * b) + c):

match (-a) * b + c   # (match (-(a * b))) + c
match (-a) * b + c:  # match (((-a) * b) + c):

match ().a   # (match()).a
match ().a:  # match (().a):

match (()).a   # (match(())).a
match (()).a:  # match (().a):

match ((),).a   # (match(())).a
match ((),).a:  # match (((),).a):

match (a).b   # (match(a)).b
match (a).b:  # match (a.b):

match (a.b)   # match(a.b)
match (a.b):  # match (a.b):

match (a,).b   # (match(a)).b     (not (match((a,))).b)
match (a,).b:  # match ((a,).b):  (not match (a.b):)

match ((a,)).b   # (match((a,))).b
match ((a,)).b:  # match ((a,).b):

match [a].b   # (match[a]).b
match [a].b:  # match ([a].b):

match [a,].b   # (match[(a,)]).b  (not (match[a]).b)
match [a,].b:  # match ([a].b):   (not match ([(a,)].b):)

match [(a,)].b   # (match[(a,)]).b
match [(a,)].b:  # match ([(a,)].b):

My rough plan (slightly amended after looking again) was going to be:

• start parsing match * …, match(EXPR)…, match[EXPR]… under the assumption that it’s an expression with a match identifier, but
• start parsing match + …, match - …, match()…, match(EXPR, …, match[EXPR, … under the assumption that it’s a match statement,

so that in either case there should be enough information to convert to the other form later if we find the initial assumption was wrong. This specific initial classification is important: there’s not enough information to translate in the other direction in either case.

Of course, this is easier said than coded in LALRPOP, and I’m not entirely sure whether LALRPOP bugs are interfering with my attempt (I already found one).

Thanks, this is extremely helpful. (I kind of assumed I was missing something important given how smoothly this went.)

If you’re further along and want to keep pushing on it, I can close and defer to you. Or if you want to collaborate on a branch that works too. Or I can keep trying here based on some of the pointers above. Whatever you prefer.

Some discussion from Parso that may or may not end up being relevant here: davidhalter/parso#138 (comment)

To what degree would getting those LALRPOP changes merge help here? Do we need to consider publishing a fork for now?

I also found this interesting on how Python 3.5 and earlier handled async and await as soft keywords via lexer tricks, but I doubt they’d help us here: https://benjam.info/blog/posts/2019-09-18-python-deep-dive-tokenizer/#async-and-await

Although… could we have the lexer look ahead? Is that crazy? Isn’t it a match statement IFF the line starts with match and ends in a colon? (Haven’t thought this through, I assume that won’t work.)

No I don’t think a lexer trick like that can work:

match()[a:
    b]
match()[a:
    b]:

Well you’d need to skip over bracketed content, no? I.e., look ahead until you hit a logical newline (not a physical newline)? At which point you’d either have a semicolon or not.

Put differently, what I’m asking is whether we could backtrack or lookahead in the lexer (where we have more control) rather than complicating the parser. E.g. I think LALRPOP does some backtracking in its own lexer (https://github.com/lalrpop/lalrpop/blob/b777a6a884052d6283971f71b3ba453aa6d417ab/lalrpop/src/tok/mod.rs#L138), but I’d have to try this myself to have a real opinion on it.

This is able to parse all of test_patma.py.

From what I can tell, performance is the ~same or within 1% (for parsing CPython via Ruff).

@andersk could you review this PR?

@youknowone I say we merge this later today, what do you think?

Totally fine if andersk is busy. I forgot the specific branch, but I found @charliermarsh already tested it with ruff somewhere.
Here: astral-sh/ruff#3047

Thanks @flying-sheep

unpinning #2759

Yeah I've run that branch over the following codebases and, in each case, verified that the lint violation counts are identical before and after that change:

• Hatch (198 violations)
• Pandas (2766 violations)
• Zulip (151 violations)
• scikit-build-core (9 violations)
• Airflow (11 violations)

On CPython, we see 18367 violations before and 18533 after; however, one fewer parse error in test_patma.py, so more violations are expected, since we're now actually analyzing that file.

Merging so we don't unnecessarily hold up astral-sh/ruff#282 too, really big thanks for this @charliermarsh!