Improper handling of Unicode Properties in character set 3.16a2 #161814

Bug report

I found this when evaluating the Unicode Properties in the recent alpha release for Re. I realize this work is a WIP, but figured I'd share in case this issue wasn't already known.

With re.IGNORECASE, a character set that contains a case-sensitive property escape (\p{Lu}, \p{Lt}, \p{Uppercase}, \p{Lowercase}, or their \P{...} forms) together with a cased literal or range gives the wrong results. Adding a member to a set can remove matches:

>>> import re
>>> re.fullmatch(r'[\p{Lu}]', 'A', re.IGNORECASE)
<re.Match object; span=(0, 1), match='A'>
>>> re.fullmatch(r'[\p{Lu}x]', 'A', re.IGNORECASE)    # expected a match
>>> re.fullmatch(r'[\p{Lu}x]', 'A')
<re.Match object; span=(0, 1), match='A'>

Below is info gathered by AI, hopefully it is helpful, but the important part is the behavior that doesn't match expectations.

More examples, all with re.IGNORECASE:

"Expected" is how these properties already behave with IGNORECASE outside a set, or as the only member of a
set: \p{Lu} and [\p{Lu}] match 'A' but not 'a'.

The same happens with an inline (?i:...), and with re.ASCII | re.IGNORECASE for ASCII characters. Sets without
a cased literal or range, such as [\p{Lu}1] and [\p{Lu}\d], are not affected.

Cause: in Lib/re/_compiler.py, when a set under IGNORECASE contains a cased literal or range (hascased
from _optimize_charset()), the whole set is compiled as IN_UNI_IGNORE (IN_IGNORE with ASCII). Those
opcodes lowercase the character before testing it against every item in the set, including CATEGORY items. The
literals in the set are already lowercased, so they still match correctly, but the categories are now tested
against the lowercased character: 'A' becomes 'a' before the Lu test.

For the categories that existed before 3.16 (\d, \s, \w), lowercasing a character never changes whether it is
in the category, so this was harmless. It does for the new CATEGORY_LU, CATEGORY_TITLE, CATEGORY_UPPER and
CATEGORY_LOWER, and their negations. Checking every code point that changes when lowercased, these are the only
supported properties whose membership changes (Lu, Lt, Uppercase, Lowercase).

A fix could test CATEGORY items against the original character rather than the lowercased one in the
ignore-case set opcodes, or have the compiler keep those categories out of the ignore-case check.

CPython versions tested on:

3.16

Operating systems tested on:

macOS