Fix string casing against the pinned Unicode database - #9824

Summary

Use the project's pinned ICU Unicode data consistently for str.lower, str.upper, str.istitle, and the lowercase/uppercase portions of capitalize, title and swapcase. Preserve the ASCII VM fast paths and WTF-8 surrogate handling.

Rust 1.100 ships Unicode 18 data, while CPython 3.15 and our Unicode database use Unicode 17. Mixing Rust's char/str case operations with ICU predicates makes test_unicodedata.UnicodeMethodsTest.test_method_checksum fail on main, independently of #9816 and #9820. For example, '\u0277'.upper() must remain unchanged with Unicode 17, but Rust's newer tables map it to U+A7DD.

The existing Unicode build script generates compact full-mapping and contextual-property tables from ICU. Runtime casing uses immutable table lookups, including one-to-many mappings, and CPython's lower_ucs4 final-sigma context rules. Lone surrogates remain context boundaries. There is no runtime cache or toolchain-version exception, and no Lib test expectations change.

A direct ICU runtime implementation was rejected after release benchmarks showed substantial Latin-text overhead. The generated-table implementation retains an ASCII-prefix path and statically dispatched character output. The tiny table/output helpers are inlined to avoid the measured call overhead.

CPython reference: Objects/unicodeobject.c, v3.15.0 (lower_ucs4, do_upper_or_lower, do_swapcase, unicode_istitle_impl).

Validation

• Reproduced main's exact checksum mismatch: a6f9fa98bde3bc2c24ab040ad9fe67b073971bbe versus 8b2615a9fc627676cbc0b6fac0191177df97ef5f.
• Unicode crate: 26 unit/integration tests passed, including all scalar values against ICU for generated lower/upper mappings and all four casing predicates; ASCII-prefix sigma/combining-mark and lone-surrogate cases included.
• Release microbenchmarks of actual helpers against Rust std, short and repeated Latin, mixed ASCII/non-ASCII, Greek, Deseret and Turkish/full-expansion inputs. Final Latin/mixed lower was 0.876–0.950 times baseline time, upper 0.856–1.002 times baseline. Other tested inputs were faster. Against the previous main helper implementations, title/capitalize/swapcase took 0.209–0.654 times baseline time on Latin, Greek, mixed and full-expansion inputs. These are microbenchmarks, not a universal speedup claim. Pure ASCII interpreter dispatch is unchanged.
• Independent source review completed.
• Commit hooks passed.
• Rebuilt the interpreter on main 9f6a743 plus this change. All 0x110000 code points matched CPython 3.15.2 for lower/upper/title/capitalize/swapcase/casefold/istitle, with additional sigma, case-ignorable and surrogate-context cases.
• -m test test_unicodedata test_str: 212 tests passed, 19 skipped.
• cargo clippy -p rustpython-unicode --all-targets -- -D warnings: passed.
• Workspace tests were attempted but the local filesystem filled while linking the call_between_rust_and_python example (ld Bus error), before aggregate tests ran. This is not recorded as a passing local aggregate run; the subsequent hosted runs below provide aggregate coverage.
• Workspace strict Clippy (--all-targets, excluding wasm/venvlauncher/capi, -D warnings): passed. Separate CAPI strict Clippy (--all-targets -- -D warnings): passed.
• Hosted CI run 43373063655 completed successfully on head 2012fb1b53d5205b806d5098a83adfb6db977a23: native Rust and Python tests on Ubuntu/macOS/Windows, WASI Python tests, WASM package, Miri, lint and remaining scheduled checks passed. Ubuntu CAPI build/tests/doctests steps were individually verified as successful. Conditional matrix/parity jobs were skipped by the workflow.

AI-assisted implementation and verification with OpenAI Codex, with independent source review. Exact model version is not exposed in this environment.