Skip to content

Share the lazy Unicode name capsule provider between codecs - #9828

Draft
youknowdot wants to merge 3 commits into
RustPython:mainfrom
youknowdot:fix/unicode-name-provider-20261113
Draft

youknowdot wants to merge 3 commits into
RustPython:mainfrom
youknowdot:fix/unicode-name-provider-20261113

Conversation

@youknowdot

@youknowdot youknowdot commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Uses the UCD private-name records from merged #9827; now based directly on main.

Implement CPython's lazy Unicode-name provider for the Unicode codecs:

  • unicodedata._ucnhash_CAPI carries an actual immutable two-callback native table, not a placeholder handle.
  • Cache the imported table per interpreter. Import failures remain retryable; successful imports survive module deletion/GC. No initialization lock is held across Python import.
  • unicode_escape acquires the provider when it encounters \N, before malformed/incomplete escape validation. namereplace checks the exception type first, then imports before inspecting its range.
  • Both codecs call the imported provider, including a valid replacement native capsule. Byte names and surrogate callback results are preserved.
  • Native name results borrow a caller-owned stack buffer, avoiding an extra String allocation.

Raw capsule creation and pointer/context mutation are now unsafe Rust APIs with explicit protocol, callback, synchronization and lifetime contracts. Their existing callers are adjusted, including C API entrypoints and Windows directory cookies. This does not change the C ABI. Merely matching a capsule name is not claimed to validate an arbitrary pointer.

Scope

Remove only test_unicodedata_unload_reload's expected-failure marker. The compiler import-failure expected failure remains: a proper Ruff runtime parse hook must also preserve literal warning/error order, speculation and recovery behavior. This PR does not attempt to work around the parser by scanning source text.

Current head and verification

Head 7a4550046b8499b86fc24e218d62fcbc8d9e8926 is rebased onto main 1d17957399ef02dab4a3ed90622d44437acc6fc1, after #9827 and #9831 merged. The provider implementation is unchanged; range-diff shows only surrounding context changed by #9820's Unicode version cleanup. No conflict-resolution behavior change was needed.

All additional Python snippet checks now live in extra_tests/snippets/stdlib_unicode.py. The separate stdlib_unicode_name_provider.py is removed and the existing stdlib_unicode_shared.py is renamed/combined. Provider failure/retry/cache checks run before other Unicode-name use. Named-escape compiler checks use runtime eval of raw source, preventing source compilation from warming the name cache before the lifecycle checks. The harness still starts one fresh interpreter for this file, including WASI; no guest subprocess, skipped assertion or platform exclusion was added.

  • Final rebased diff: all applicable pre-commit hooks passed (including Rustfmt, Ruff and cspell); whitespace checks passed.
  • Consolidated snippet: both pytest harness cases passed using CPython 3.15.2 and the existing provider-enabled RustPython binary. This RustPython binary predates the final allocation-only adjustment/rebase; it is not an exact-head rebuild.
  • A full extra_tests pytest run was attempted during consolidation: 184 passes and 13 failures were observed before stopping at the unrelated CPython concurrent GC/import stress test. It was not a full passing run.
  • Exact-head native aggregate and WASI runtime validation have not been rerun locally. New hosted CI is pending; historical results below are not claimed for this rebased head.

Historical implementation validation

  • Provider-enabled VM: UnicodeMiscTest ran 12 tests with one existing skip and the remaining compiler expected failure; UnicodeEscapeTest passed 15 tests.
  • Final native provider tests passed: foreign callbacks, non-UTF8 input/surrogate output, interpreter-local caching, and eight name-buffer boundary cases.
  • Strict Clippy for root/VM/common/stdlib with all targets passed; C API cargo check passed. Independent safety/CPython-parity review completed.
  • The earlier two-file WASI harness fix passed build, snippets and CPython unittest execution in run 43398660568, job 131796417580. The consolidated single-file version still needs its new hosted WASI result.
  • A separate non-threaded package-only Clippy invocation encountered the existing _ctypes/base.rs Arc<BufferWrite> lint; the normal root-package threading configuration passed. No unrelated suppression or ctypes change is included.

@coderabbitai

coderabbitai Bot commented Nov 13, 2026

Copy link
Copy Markdown
Contributor

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

  • Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@github-actions

Copy link
Copy Markdown
Contributor

📦 Library Dependencies

The following Lib/ modules were modified. Here are their dependencies:

[ ] test: cpython/Lib/test/test_unicodedata.py (TODO: 2)
[x] test: cpython/Lib/test/test_unicode_file.py
[x] test: cpython/Lib/test/test_unicode_file_functions.py
[x] test: cpython/Lib/test/test_unicode_identifiers.py
[x] test: cpython/Lib/test/test_ucn.py (TODO: 1)

dependencies:

dependent tests: (no tests depend on unicode)

Legend:

  • [+] path exists in CPython
  • [x] up-to-date, [ ] outdated

Run provider lifecycle checks before Unicode-name use, and compile named escapes at runtime to preserve fresh-cache coverage without subprocesses.

Assisted-by: Codex:model-identifier-unavailable
@youknowdot
youknowdot force-pushed the fix/unicode-name-provider-20261113 branch from 640ab1c to 7a45500 Compare November 14, 2026 06:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant