Skip to content

Align Linux locale collation with Python 3.15 - #9674

Merged
youknowone merged 1 commit into
RustPython:mainfrom
youknowdot:locale-wide-collation-group-20261103
Nov 4, 2026
Merged

youknowone merged 1 commit into
RustPython:mainfrom
youknowdot:locale-wide-collation-group-20261103

Conversation

@youknowdot

Copy link
Copy Markdown
Contributor

Summary

  • Route Linux _locale.strcoll and strxfrm through wide-character collation, retaining lone surrogates and rejecting embedded NULs through a shared converter after normal argument binding.
  • Encode transformation weights with CPython 3.15's exact > 0x10000 boundary; preserve individual surrogate weights and propagate libc errors separately from allocation failures.
  • Bound the second-pass result by the allocated buffer. Existing host APIs and non-Linux routes remain unchanged.

The current narrow UTF-8 route reverses comparisons in non-UTF-8 locales. In installed sv_SE, comparing z with ä returns positive through the current narrow call but negative through CPython's wide call. Reference probes found 24/100 ordering differences in each of en_US and de_DE, and 36/100 in sv_SE.

Reference: CPython 3.15 collation implementation.

Validation

  • Fresh package-clean cargo check for host_env, vm and stdlib, with rustpython-stdlib/threading, locked/offline, one job.
  • Strict Clippy for the same three packages and feature selection (-D warnings).
  • Bounded CPython 3.15 reference probes: 78 raw-wide-key comparisons and 1,014 pair-order comparisons across six installed locales; additional non-UTF-8 ordering probes, surrogate/boundary cases, NUL/type precedence and subclass callbacks.
  • rustfmt, diff check and normal pre-commit hooks.

Native RustPython execution, full suites and other platforms were not run. No tests, standard-library Python files or workflows are changed. No >Unicode raw weights or invalid-UTF-8 narrow keys were observed; allocation-failure and concurrent-size-change branches were source-reviewed, not fault-injected.

AI assistance

Implementation, source review and validation were AI-assisted using OpenAI Codex (model version not exposed).

Use wide collation strings, retain surrogate code points, encode transform weights in Python 3.15 order, and preserve errno and allocation failures.

AI-assisted implementation, source review, and validation using OpenAI Codex.

Assisted-by: OpenAI Codex:model version not exposed
@coderabbitai

coderabbitai Bot commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 3 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used all 10 included reviews currently available.

Learn how review limits work.

Review configuration:

Run configuration
Commits

Reviewing files that changed from the base of the PR and between 0b081b0 and 00c4058.

Files selected for processing: 2
  • crates/host_env/src/locale.rs
  • crates/stdlib/src/locale.rs
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@codspeed

codspeed Bot commented Nov 4, 2026

Copy link
Copy Markdown
Contributor

Merging this PR will not alter performance

✅ 62 untouched benchmarks
⏩ 4 skipped benchmarks1


Comparing youknowdot:locale-wide-collation-group-20261103 (00c4058) with main (0b081b0)

Open in CodSpeed

Footnotes

  1. 4 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

@youknowone
youknowone marked this pull request as ready for review November 4, 2026 13:08
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 4, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-04T13:14:51.064734Z 00c4058 Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@youknowone
youknowone enabled auto-merge (squash) November 4, 2026 13:08

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 00c405870f

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

buff
}

#[cfg(target_os = "linux")]

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Add the required AI-assistance commit trailer

The commit message explicitly states that implementation, source review, and validation were AI-assisted, but it contains no Assisted-by: trailer. AGENTS.md requires an Assisted-by: AGENT_NAME:MODEL_VERSION trailer for each AI tool; the free-form “AI assistance” section does not satisfy that attribution requirement. Add the required structured trailer to the commit message.

AGENTS.md reference: AGENTS.md:L23-L23

Useful? React with 👍 / 👎.

@youknowone
youknowone merged commit dc84268 into RustPython:main Nov 4, 2026
29 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants