You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Implement unicodedata XID and block properties - #9458
Generate a compact block table using the existing Unicode build machinery and the unmodified official Unicode 17.0.0 Blocks.txt. No new dependency. All 346 ranges/names match CPython's pinned generated table.
Follow CPython's single-codepoint conversion, string-subclass acceptance, type/length diagnostics, surrogate behavior, and No_Block fallback. Keep these APIs absent on ucd_3_2_0.
Run scripts/update_lib migrate against the complete pinned test_unicodedata.py; retire exactly four proven expectedFailure markers, retaining every unrelated local adaptation. No hand-edited tests or duplicate snippets.
On native-source base 698c196b2e188c46772f8a3cf76164227282a645 with Rust 1.99:
Locked/offline scoped cargo check and cargo clippy -- -D warnings for rustpython-stdlib, features compiler,threading,host_env,doc,ucd_3_2_0: passed.
CLI build with threading,stdlib,stdio,importlib,host_env,doc,ucd_3_2_0: passed. Initial bounded build timed out; one fresh cached resume completed successfully.
The four unmodified pinned CPython methods (test_isxidstart, test_isxidcontinue, test_block, test_block_invalid_input) pass on both the pinned CPython build and this RustPython build, with no skip/expected failure.
A bounded 63-case argument/value oracle per interpreter produces identical JSON, including underscores, unassigned code points, surrogates, block boundaries, No_Block, bad lengths/types, string subclasses, and legacy API absence.
All applicable normal pre-commit hooks passed, including cspell (two canonical API names added to its existing dictionary).
Final migrated methods are AST-identical to the tested canonical methods; native source is unchanged after the build.
Limits: this does not claim whole-module or whole-workspace test completion. Existing shared argument-binding arity/keyword error wording is not changed: rejection and TypeError are checked there, not exact METH_O wording. Unrelated legacy UCD/normalization/decomposition issues remain. Latest-main integration is left to PR merge CI; the exact local validation base is stated above.
AI assistance: implementation and validation prepared with OpenAI Codex.
Summary by CodeRabbit
New Features
Added unicodedata.isxidstart() and unicodedata.isxidcontinue() to check whether a character can start or continue a valid Unicode identifier.
Added unicodedata.block() to return the Unicode block name for a character, or No_Block when it belongs to no named block.
These functions require a string containing exactly one Unicode code point and report errors for invalid input.
Follow CPython 3.15 character conversion and Unicode 17 block data; retire four verified test markers via update_lib.
Assisted-by: OpenAI Codex:GPT-6.1
Reviewing files that changed from the base of the PR and between bcaa07a and 3d40a27.
Files ignored due to path filters: 1
Lib/test/test_unicodedata.py is excluded by !Lib/**
Files selected for processing: 6
.cspell.json
crates/stdlib/src/unicodedata.rs
crates/unicode/build.rs
crates/unicode/src/data.rs
crates/unicode/unicode/README.md
crates/unicode/unicode/latest/Blocks.txt
Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 8 remain after this review.
Walkthrough
The change adds Unicode block data generation and lookup, and exposes unicodedata.isxidstart, unicodedata.isxidcontinue, and unicodedata.block. The new functions validate that their argument is a string containing exactly one code point.
Changes: Unicode property and block APIs
Layer / File(s)
Summary
Unicode block data and lookup crates/unicode/unicode/latest/Blocks.txt, crates/unicode/build.rs, crates/unicode/src/data.rs, crates/unicode/unicode/README.md
Adds Unicode 17.0.0 block ranges, generates a block table from Blocks.txt, and adds a lookup that returns the matching block name or No_Block. The README identifies the data source and version.
RustPython/RustPython#6839 — The new unicodedata APIs support the issue’s objective of syncing Python 3.15 library behavior; the provided summary does not show the related bundled-library or test updates.
Suggested reviewers:youknowone
Merge Risk: Minimal · up to 3d40a
The new Unicode APIs reject keyword arguments as required. No actionable merge-blocking issue remains; merge after normal checks.
Pre-merge checks: 5
Check name
Status
Explanation
Description Check
Check skipped - CodeRabbit’s high-level summary is enabled.
Title check
The title clearly and concisely describes the main changes: adding Unicode XID and block properties to unicodedata.
Docstring Coverage
Docstring coverage is 80.00% which is sufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentati…
Linked Issues check
Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check
Check skipped because no linked issues were found for this pull request.
Full details: Docstring Coverage
Explanation
Docstring coverage is 80.00% which is sufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentation-only edits and deletions within surviving functions are included; unchanged diff context and fully deleted functions are excluded. Analyzed 5 functions across 3 files. (3 skipped: 3 unsupported.)
Finishing Touches
🧪 Generate unit tests (beta)
Create a new PR
Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
Warning
Some tools did not complete. Review the errors below.
Biome (2.5.15)
.cspell.json
File contains syntax errors that prevent linting: Line 1: Expected an array, an object, or a literal but instead found '// See: https://github.com/streetsidesoftware/cspell/tree/; Line 6: Expected an array, an object, or a literal but instead found '// "@cspell/dict-cpp/cspell-ext.json",'.; Line 2: End of file expected; Line 6: End of file expected; Line 7: End of file expected; Line 7: End of file expected; Line 8: End of file expected; Line 8: End of file expected; Line 9: End of file expected; Line 9: End of file expected; Line 10: End of file expected; Line 10: End of file expected; Line 12: End of file expected; Line 12: End of file expected; Line 12: End of file expected; Line 12: End of file expected; Line 14: End of file expected; Line 14: End of file expected; Line 14: End of file expected; Line 14: End of file expected; Line 16: End of file expected; Line 16: End of file expected; Line 17: Expected an array, an object, or a literal but instead found '// Sometimes keeping same
... [truncated 1397 characters] ...
; Line 130: End of file expected; Line 131: End of file expected; Line 131: End of file expected; Line 132: End of file expected; Line 132: End of file expected; Line 133: End of file expected; Line 133: End of file expected; Line 134: End of file expected; Line 134: End of file expected; Line 135: End of file expected; Line 135: End of file expected; Line 137: End of file expected; Line 138: End of file expected; Line 140: End of file expected; Line 140: End of file expected; Line 140: End of file expected; Line 141: End of file expected; Line 142: End of file expected; Line 142: End of file expected; Line 142: End of file expected; Line 143: End of file expected; Line 145: End of file expected; Line 145: End of file expected; Line 145: End of file expected; Line 151: End of file expected
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Implement Python 3.15's module-level
unicodedata.isxidstart(),isxidcontinue(), andblock().No_Blockfallback. Keep these APIs absent onucd_3_2_0.scripts/update_lib migrateagainst the complete pinnedtest_unicodedata.py; retire exactly four provenexpectedFailuremarkers, retaining every unrelated local adaptation. No hand-edited tests or duplicate snippets.CPython basis
Reference: d220409a13b77de7a6c30abe16ea90db01528edc.
Validation
On native-source base
698c196b2e188c46772f8a3cf76164227282a645with Rust 1.99:cargo checkandcargo clippy -- -D warningsforrustpython-stdlib, featurescompiler,threading,host_env,doc,ucd_3_2_0: passed.threading,stdlib,stdio,importlib,host_env,doc,ucd_3_2_0: passed. Initial bounded build timed out; one fresh cached resume completed successfully.test_isxidstart,test_isxidcontinue,test_block,test_block_invalid_input) pass on both the pinned CPython build and this RustPython build, with no skip/expected failure.No_Block, bad lengths/types, string subclasses, and legacy API absence.Limits: this does not claim whole-module or whole-workspace test completion. Existing shared argument-binding arity/keyword error wording is not changed: rejection and
TypeErrorare checked there, not exact METH_O wording. Unrelated legacy UCD/normalization/decomposition issues remain. Latest-main integration is left to PR merge CI; the exact local validation base is stated above.AI assistance: implementation and validation prepared with OpenAI Codex.
Summary by CodeRabbit
unicodedata.isxidstart()andunicodedata.isxidcontinue()to check whether a character can start or continue a valid Unicode identifier.unicodedata.block()to return the Unicode block name for a character, orNo_Blockwhen it belongs to no named block.