Align Unicode lookup and decomposition with CPython 3.15 by youknowdot · Pull Request #9476 · RustPython/RustPython

youknowdot · 2026-10-30T07:15:44Z

Summary

Validate names returned by unicode_names2 against canonical names, while preserving exact case-insensitive alias and named-sequence lookup. Reject loose whitespace/hyphen matching and noncanonical CJK hexadecimal names.
Match CPython's distinction for the old UCD view: lookup() rejects aliases/sequences but accepts ordinary names from the shared database; name() and decomposition() retain historical membership checks.
Return full Hangul L/V/T decomposition and use current decomposition mappings in both views, removing the incorrect historical normalization-correction override. Normalization code is unchanged.
Use scripts/update_lib migrate to retire four related test_unicodedata markers while preserving the other nine adaptations.

Sources: pinned CPython lookup, decomposition, and canonical tests.

Validation and limits

Scoped locked/offline CLI check, strict Clippy (-D warnings) and build succeeded.
120 bounded scratch observations matched pinned CPython exactly, including exception text, aliases/sequences, significant hyphens, both UCD views, Hangul and corrected decomposition mappings.
Four restored test_unicodedata methods passed in both interpreters. Of twelve selected canonical tests, CPython passed twelve; RustPython passed eleven and retained one pre-existing expected failure in test_ucn.test_tangut_ideographs. The strict wrapper therefore exited 1, not a full-suite pass.
A separately bounded single-test diagnostic confirmed that expected failure is the existing ast.literal_eval/compile Unicode-name literal SyntaxError, not unicodedata.lookup. Its decorator and test body remain unchanged; direct Tangut lookup is covered by the passing differential observations.

No full Unicode sweep, full workspace/CAPI suite, normalization corpus or cross-platform runtime validation was performed. This does not claim full Unicode or parser compatibility.

AI assistance: OpenAI Codex (model version unavailable), disclosed in the commit trailer. Automated validation is not human review.

Validate Unicode names instead of accepting loose dependency matching, preserve old-UCD alias and sequence restrictions without filtering ordinary names, and match CPython's Hangul and historical-membership decomposition behavior. Remove the incorrect historical correction override and refresh four related unicodedata test markers using update_lib. Leave normalization unchanged. Assisted-by: OpenAI Codex: model version unavailable

coderabbitai · 2026-10-30T07:15:51Z

Caution

Review failed

The pull request is closed.

Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

github-actions · 2026-10-30T07:16:54Z

📦 Library Dependencies

The following Lib/ modules were modified. Here are their dependencies:

[x] lib: cpython/Lib/weakref.py
[x] lib: cpython/Lib/_weakrefset.py
[x] test: cpython/Lib/test/test_weakref.py (TODO: 7)
[x] test: cpython/Lib/test/test_weakset.py

dependencies:

weakref

dependent tests: (219 tests)

[x] test: cpython/Lib/test/test_dict.py (TODO: 1)
[x] test: cpython/Lib/test/test_dictcomps.py
[x] test: cpython/Lib/test/test_dictviews.py (TODO: 1)
[x] test: cpython/Lib/test/test_userdict.py
[x] test: cpython/Lib/test/mapping_tests.py

dependencies:

dependent tests: (no tests depend on dict)

[ ] test: cpython/Lib/test/test_free_threading (TODO: 1)

dependencies:

dependent tests: (no tests depend on free_threading)

[x] test: cpython/Lib/test/test_grp.py (TODO: 1)

dependencies:

dependent tests: (195 tests)

[x] lib: cpython/Lib/imaplib.py
[x] test: cpython/Lib/test/test_imaplib.py (TODO: 1)

dependencies:

imaplib

dependent tests: (1 tests)

imaplib: test_imaplib

[ ] test: cpython/Lib/test/test_capi

dependencies:

dependent tests: (no tests depend on capi)

[x] lib: cpython/Lib/functools.py
[ ] test: cpython/Lib/test/test_functools.py (TODO: 1)

dependencies:

functools

dependent tests: (53 tests)

functools: test_annotationlib test_argparse test_asyncio test_builtin test_bytes test_call test_clinic test_codeop test_context test_contextlib_async test_ctypes test_curses test_datetime test_dbm_dumb test_dbm_sqlite3 test_dis test_doctest test_enum test_fractions test_free_threading test_functools test_genericalias test_gettext test_gzip test_hmac test_inspect test_io test_iter test_itertools test_math_property test_monitoring test_posixpath test_profiling test_pyexpat test_pyrepl test_random test_repl test_reprlib test_shutil test_signal test_sort test_sqlite3 test_ssl test_sys_settrace test_timeout test_tkinter test_traceback test_typing test_unicodedata test_unittest test_winsound test_xml_etree test_zoneinfo

[x] lib: cpython/Lib/enum.py
[x] test: cpython/Lib/test/test_enum.py (TODO: 1)

dependencies:

enum

dependent tests: (18 tests)

enum: test_argparse test_ast test_enum test_httplib test_json test_opcache test_patma test_pstats test_pydoc test_signal test_socket test_ssl test_str test_time test_tkinter test_types test_typing test_uuid

[x] lib: cpython/Lib/sqlite3
[x] test: cpython/Lib/test/test_sqlite3 (TODO: 19)

dependencies:

sqlite3

dependent tests: (2 tests)

sqlite3: test_dbm_sqlite3 test_sqlite3

[ ] lib: cpython/Lib/io.py
[x] lib: cpython/Lib/_pyio.py
[ ] test: cpython/Lib/test/test_io (TODO: 10)

dependencies:

io (native: _io, _thread, errno, msvcrt, sys)
- _pyio
- _collections_abc, abc, codecs, locale, os, stat, warnings

dependent tests: (105 tests)

io: test__colorize test_android test_argparse test_ast test_asyncio test_base64 test_buffer test_builtin test_bz2 test_calendar test_cmd test_cmd_line_script test_codecs test_compile test_compileall test_compiler_assemble test_concurrent_futures test_configparser test_contextlib test_csv test_dbm_dumb test_descr test_dis test_email test_enum test_fileinput test_free_threading test_ftplib test_generated_cases test_getpass test_gzip test_hashlib test_http_cookiejar test_httplib test_httpservers test_importlib test_inspect test_io test_json test_logging test_lzma test_mailbox test_marshal test_memoryview test_mimetypes test_minidom test_multibytecodec test_optparse test_pathlib test_pdb test_peg_generator test_pickle test_pickletools test_platform test_plistlib test_pprint test_print test_profile test_profiling test_pstats test_pty test_pulldom test_pydoc test_pyexpat test_pyrepl test_quopri test_regrtest test_remote_pdb test_robotparser test_sax test_shlex test_shutil test_site test_smtplib test_socket test_socketserver test_subprocess test_support test_sys test_tarfile test_tempfile test_threadedtempfile test_timeit test_tokenize test_traceback test_types test_typing test_unittest test_urllib test_urllib2 test_uuid test_wave test_webbrowser test_winconsoleio test_wsgiref test_xml_dom_xmlbuilder test_xml_etree test_xml_etree_c test_xmlrpc test_xpickle test_zipapp test_zipfile test_zipimport test_zoneinfo test_zstd

[ ] test: cpython/Lib/test/test_pyrepl (TODO: 5)
[x] test: cpython/Lib/test/test_repl.py (TODO: 3)

dependencies:

dependent tests: (no tests depend on pyrepl)

[x] lib: cpython/Lib/xml
[x] test: cpython/Lib/test/test_xml_etree.py (TODO: 18)
[ ] test: cpython/Lib/test/test_xml_etree_c.py
[x] test: cpython/Lib/test/test_minidom.py (TODO: 2)
[x] test: cpython/Lib/test/test_pulldom.py
[ ] test: cpython/Lib/test/test_pyexpat.py (TODO: 41)
[x] test: cpython/Lib/test/test_sax.py (TODO: 25)
[x] test: cpython/Lib/test/test_xml_dom_minicompat.py
[x] test: cpython/Lib/test/test_xml_dom_xmlbuilder.py

dependencies:

xml

dependent tests: (39 tests)

xml: test_doctest test_minidom test_pulldom test_pydoc test_pyexpat test_regrtest test_sax test_typing test_xml test_xml_dom_minicompat test_xml_dom_xmlbuilder test_xml_etree
- plistlib: test_plistlib
  - platform: test__locale test__osx_support test_android test_asyncio test_baseexception test_builtin test_calendar test_cmath test_ctypes test_fcntl test_math test_mimetypes test_os test_platform test_shutil test_socket test_ssl test_startfile test_strptime test_sysconfig test_time test_tkinter test_winreg test_wsgiref
  - webbrowser: test_webbrowser
- xmlrpc.client: test_xmlrpc

[x] lib: cpython/Lib/test/libregrtest

dependencies:

regrtest

dependent tests: (no tests depend on regrtest)

[x] lib: cpython/Lib/re
[ ] test: cpython/Lib/test/test_re.py (TODO: 3)
[x] test: cpython/Lib/test/re_tests.py

dependencies:

re

dependent tests: (86 tests)

re: test_android test_ast test_asyncio test_binascii test_builtin test_bytes test_clinic test_cmd_line test_ctypes test_dict test_dis test_docxmlrpc test_dtrace test_email test_embed test_faulthandler test_filecmp test_fileinput test_fnmatch test_format test_frame test_free_threading test_fstring test_functools test_future_stmt test_genericalias test_glob test_hashlib test_http_cookiejar test_httplib test_httpservers test_imaplib test_importlib test_inspect test_ipaddress test_launcher test_logging test_mailbox test_mmap test_optparse test_ordered_dict test_pdb test_pprint test_profiling test_pydoc test_pyexpat test_pyrepl test_re test_regrtest test_remote_pdb test_runpy test_set test_site test_smtplib test_socket test_ssl test_strftime test_string test_strptime test_strtod test_structseq test_symtable test_syntax test_sys test_sysconfig test_tarfile test_tempfile test_tkinter test_tokenize test_tools test_traceback test_turtle test_types test_typing test_unittest test_unparse test_venv test_webbrowser test_winapi test_with test_wsgiref test_xmlrpc test_zipfile test_zipimport test_zoneinfo test_zstd

[x] lib: cpython/Lib/profiling
[x] test: cpython/Lib/test/test_profiling

dependencies:

profiling

dependent tests: (3 tests)

profiling: test_profiling
- cProfile: test_free_threading test_pstats

[x] lib: cpython/Lib/codecs.py
[x] test: cpython/Lib/test/test_charmapcodec.py
[x] test: cpython/Lib/test/test_codeccallbacks.py (TODO: 3)
[x] test: cpython/Lib/test/test_codecencodings_cn.py
[x] test: cpython/Lib/test/test_codecencodings_hk.py
[x] test: cpython/Lib/test/test_codecencodings_iso2022.py
[x] test: cpython/Lib/test/test_codecencodings_jp.py
[x] test: cpython/Lib/test/test_codecencodings_kr.py
[x] test: cpython/Lib/test/test_codecencodings_tw.py
[x] test: cpython/Lib/test/test_codecmaps_cn.py
[x] test: cpython/Lib/test/test_codecmaps_hk.py
[x] test: cpython/Lib/test/test_codecmaps_jp.py
[x] test: cpython/Lib/test/test_codecmaps_kr.py
[x] test: cpython/Lib/test/test_codecmaps_tw.py
[x] test: cpython/Lib/test/test_codecs.py (TODO: 2)
[x] test: cpython/Lib/test/test_multibytecodec.py
[x] test: cpython/Lib/test/testcodec.py

dependencies:

codecs

dependent tests: (200 tests)

[x] lib: cpython/Lib/typing.py
[ ] test: cpython/Lib/test/test_typing.py (TODO: 1)
[x] test: cpython/Lib/test/test_type_aliases.py
[x] test: cpython/Lib/test/test_type_annotations.py
[x] test: cpython/Lib/test/test_type_params.py
[x] test: cpython/Lib/test/test_genericalias.py

dependencies:

typing

dependent tests: (22 tests)

typing: test_annotationlib test_builtin test_copy test_descr test_enum test_fractions test_free_threading test_funcattrs test_functools test_genericalias test_grammar test_inspect test_isinstance test_patma test_peg_generator test_pydoc test_pyrepl test_remote_pdb test_type_aliases test_type_params test_types test_typing

[ ] test: cpython/Lib/test/test_metaclass.py (TODO: 4)

dependencies:

dependent tests: (no tests depend on metaclass)

[ ] test: cpython/Lib/test/test_with.py

dependencies:

dependent tests: (no tests depend on with)

[ ] test: cpython/Lib/test/test_cmd_line.py (TODO: 18)

dependencies:

dependent tests: (no tests depend on cmd_line)

[ ] test: cpython/Lib/test/test_faulthandler.py (TODO: 4)

dependencies:

dependent tests: (5 tests)

faulthandler: test_cmd_line test_concurrent_futures test_faulthandler test_inspect test_xxtestfuzz

[ ] test: cpython/Lib/test/test_unicodedata.py (TODO: 7)
[x] test: cpython/Lib/test/test_unicode_file.py
[x] test: cpython/Lib/test/test_unicode_file_functions.py
[x] test: cpython/Lib/test/test_unicode_identifiers.py
[x] test: cpython/Lib/test/test_ucn.py (TODO: 1)

dependencies:

dependent tests: (no tests depend on unicode)

[x] lib: cpython/Lib/_pylong.py
[ ] test: cpython/Lib/test/test_int.py (TODO: 2)
[x] test: cpython/Lib/test/test_long.py (TODO: 4)
[x] test: cpython/Lib/test/test_int_literal.py

dependencies:

int

dependent tests: (no tests depend on int)

[ ] test: cpython/Lib/test/test_set.py (TODO: 1)

dependencies:

dependent tests: (no tests depend on set)

[ ] test: cpython/Lib/test/test_bytes.py (TODO: 11)

dependencies:

dependent tests: (no tests depend on bytes)

[x] lib: cpython/Lib/compileall.py
[x] test: cpython/Lib/test/test_compileall.py (TODO: 2)

dependencies:

compileall

dependent tests: (1 tests)

compileall: test_compileall

[x] lib: cpython/Lib/collections
[x] lib: cpython/Lib/_collections_abc.py
[x] test: cpython/Lib/test/test_collections.py
[x] test: cpython/Lib/test/test_deque.py (TODO: 1)
[x] test: cpython/Lib/test/test_defaultdict.py
[x] test: cpython/Lib/test/test_ordered_dict.py

dependencies:

collections

dependent tests: (378 tests)

[x] test: cpython/Lib/test/test_sys.py (TODO: 4)
[x] test: cpython/Lib/test/test_syslog.py
[x] test: cpython/Lib/test/test_sys_setprofile.py (TODO: 1)
[x] test: cpython/Lib/test/test_sys_settrace.py
[x] test: cpython/Lib/test/test_audit.py (TODO: 16)
[x] test: cpython/Lib/test/audit-tests.py

dependencies:

dependent tests: (250 tests)

sys: regrtestdata test___all__ test__colorize test__locale test__osx_support test_android test_annotationlib test_argparse test_array test_asdl_parser test_ast test_asyncio test_audit test_base64 test_bdb test_bigaddrspace test_bigmem test_binascii test_bisect test_buffer test_builtin test_bytes test_bz2 test_c_locale_coercion test_calendar test_capi test_class test_clinic test_cmath test_cmd test_cmd_line test_cmd_line_script test_code test_code_module test_codeccallbacks test_codecs test_collections test_compile test_compileall test_complex test_concurrent_futures test_context test_contextlib test_coroutines test_crossinterp test_csv test_ctypes test_curses test_datetime test_dbm test_dbm_sqlite3 test_decimal test_descr test_dict test_difflib test_dis test_doctest test_docxmlrpc test_dtrace test_dynamic test_dynamicclassattribute test_email test_embed test_ensurepip test_enum test_enumerate test_eof test_epoll test_except_star test_exceptions test_external_inspection test_faulthandler test_fcntl test_file_eintr test_fileinput test_float test_fork1 test_format test_fractions test_frame test_free_threading test_frozen test_functools test_future_stmt test_gc test_generated_cases test_generators test_genericpath test_genexps test_getopt test_glob test_grammar test_gzip test_hash test_hashlib test_http_cookiejar test_httpservers test_importlib test_inspect test_int test_interpreters test_io test_ioctl test_iter test_itertools test_json test_launcher test_list test_listcomps test_locale test_logging test_long test_lzma test_mailbox test_marshal test_math test_memoryview test_metaclass test_mimetypes test_mmap test_monitoring test_msvcrt test_multibytecodec test_multiprocessing_forkserver test_multiprocessing_main_handling test_netrc test_ntpath test_numeric_tower test_opcache test_operator test_optparse test_ordered_dict test_os test_osx_env test_pathlib test_patma test_pdb test_peepholer test_perfmaps test_pickle test_pkg test_pkgutil test_platform test_plistlib test_popen test_posixpath test_print test_profile test_profiling test_property test_pty test_pwd test_py_compile test_pyclbr test_pydoc test_pyexpat test_pyrepl test_quopri test_raise test_range test_re test_readline test_regrtest test_remote_pdb test_repl test_reprlib test_resource test_runpy test_sax test_scope test_script_helper test_select test_selectors test_shutil test_signal test_site test_slice test_smtplib test_socket test_source_encoding test_sqlite3 test_ssl test_stable_abi_ctypes test_startfile test_stat test_statistics test_str test_strftime test_string_literals test_strptime test_strtod test_struct test_subprocess test_support test_sys test_sys_setprofile test_sys_settrace test_sysconfig test_syslog test_tabnanny test_tarfile test_tcl test_tempfile test_termios test_threading test_threading_local test_threadsignals test_time test_timeit test_tkinter test_tomllib test_tools test_trace test_traceback test_tracemalloc test_ttk test_tuple test_type_cache test_type_comments test_types test_typing test_unicode_file test_unicode_file_functions test_unicodedata test_unittest test_urllib test_urllib2 test_urllib2net test_urlparse test_utf8_mode test_uuid test_venv test_wait3 test_wait4 test_wave test_weakref test_webbrowser test_winconsoleio test_winreg test_with test_wsgiref test_xml_etree test_xmlrpc test_xpickle test_zipapp test_zipfile test_zipfile64 test_zipimport test_zipimport_support test_zlib

Legend:

[+] path exists in CPython
[x] up-to-date, [ ] outdated

codspeed · 2026-10-30T07:41:18Z

Merging this PR will not alter performance

✅ 62 untouched benchmarks
⏩ 4 skipped benchmarks¹

_{Comparing youknowdot:python315-unicode-lookup-decomposition-20261030 (d194a68) with main (cc18267)}

4 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

Match CPython's old-UCD width policy: retain modern width defaults for characters still unassigned, and use the historical table for assigned characters. Restore the bounded east Asian width tests and retire the obsolete 9.0 override through update_lib. Permit explicit exact-source retirement of unconditional skip markers in the existing updater path, preserving its exact-match and consumed-once guards. Add focused tool tests for successful and rejected retirements. Assisted-by: OpenAI Codex: model version unavailable

youknowone

review this again if it is aligned to 3.2.0 feature well

youknowdot · 2026-10-30T15:29:50Z

In response to the 3.2 feature review:

I rechecked the 3.2 feature against pinned CPython source and our actual old data. #9476 targets CPython's ucd_3_2_0 API, not a literal historical database for every operation:

Old lookup accepts current ordinary names but rejects aliases/sequences; unlike name, it does not check old membership. The lookup correction preserves this asymmetry.
Old decomposition rejects newly assigned characters, then uses current mappings/full Hangul decomposition. Thus U+2F868 gives 36FC; historical 2136A belongs to the separate old normalization corrections, not this API.
EAW distinguishes newly assigned from still-unassigned characters: old-view U+9FA6 is N, but still-unassigned U+3FFFD remains W. The generator leaves current width defaults for the latter. Our historical derived-width file explicitly omits unassigned points.

The earlier scoped comparisons covered 120 lookup/decomposition rows and 100 EAW/property rows across both views, on their respective implementation heads. This review additionally checked 27 finite source-table boundaries, not an exhaustive sweep or new runtime run.

Importantly, RustPython's pre-existing normalize/is_normalized still ignore the old view. Those functions and feature wiring are unchanged; this PR does not complete all Unicode 3.2 semantics.

Consolidate the generic marker-retirement helper and regression coverage in RustPython#9578. Preserve the Unicode implementation and whole-file test changes unchanged. Assisted-by: OpenAI Codex:model version not exposed

chatgpt-codex-connector · 2026-11-06T12:08:40Z

Codex Review Summary: ❌ Failed

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	❌ Failed 2026-11-06T12:08:35.811858Z We couldn’t complete this review. Comment `@codex review` to try again.	`f6b8464`	Draft marked ready

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

chatgpt-codex-connector · 2026-11-06T12:08:42Z

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.

youknowone approved these changes Oct 30, 2026

View reviewed changes

youknowone requested changes Oct 30, 2026

View reviewed changes

youknowdot force-pushed the python315-unicode-lookup-decomposition-20261030 branch from 8cb31d7 to d194a68 Compare November 3, 2026 07:13

youknowdot mentioned this pull request Nov 3, 2026

Accept bytes in Unicode name lookup #9585

Merged

Keep Unicode changes independent of updater tooling

f6b8464

Consolidate the generic marker-retirement helper and regression coverage in RustPython#9578. Preserve the Unicode implementation and whole-file test changes unchanged. Assisted-by: OpenAI Codex:model version not exposed

youknowdot mentioned this pull request Nov 5, 2026

Consolidate whole-file update_lib migration guards #9578

Merged

youknowone approved these changes Nov 6, 2026

View reviewed changes

youknowone marked this pull request as ready for review November 6, 2026 12:08

youknowone merged commit 83ba227 into RustPython:main Nov 6, 2026
20 of 21 checks passed

Navigation Menu

Repository navigation

Align Unicode lookup and decomposition with CPython 3.15 - #9476

Align Unicode lookup and decomposition with CPython 3.15#9476
youknowone merged 3 commits into
RustPython:mainfrom
youknowdot:python315-unicode-lookup-decomposition-20261030

youknowdot commented Oct 30, 2026

Uh oh!

coderabbitai Bot commented Oct 30, 2026 •

edited

Loading

Review failed

Uh oh!

github-actions Bot commented Oct 30, 2026 •

edited

Loading

Uh oh!

codspeed Bot commented Oct 30, 2026 •

edited

Loading

Uh oh!

youknowone left a comment

Uh oh!

youknowdot commented Oct 30, 2026

Uh oh!

chatgpt-codex-connector Bot commented Nov 6, 2026

Uh oh!

chatgpt-codex-connector Bot commented Nov 6, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

youknowdot commented Oct 30, 2026

Summary

Validation and limits

Uh oh!

coderabbitai Bot commented Oct 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review failed

Uh oh!

github-actions Bot commented Oct 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

📦 Library Dependencies

Uh oh!

codspeed Bot commented Oct 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merging this PR will not alter performance

Footnotes

Uh oh!

youknowone left a comment

Choose a reason for hiding this comment

Uh oh!

youknowdot commented Oct 30, 2026

Uh oh!

chatgpt-codex-connector Bot commented Nov 6, 2026

Uh oh!

chatgpt-codex-connector Bot commented Nov 6, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

coderabbitai Bot commented Oct 30, 2026 •

edited

Loading

github-actions Bot commented Oct 30, 2026 •

edited

Loading

codspeed Bot commented Oct 30, 2026 •

edited

Loading