Add more unicode functions to the c-api by bschoenmaeckers · Pull Request #8272 · RustPython/RustPython

bschoenmaeckers · 2026-07-13T13:22:04Z

Summary by CodeRabbit

New Features
- Extended Unicode C-API with new constructors for UTF-8 strings, objects, and ordinals, plus default encoding reporting and string interning.
- Added Unicode encoding helpers (ASCII, Latin-1, escape variants, UTF-16/UTF-32) and decoding APIs (general and encoding-specific wrappers).
- Added Unicode operations and comparisons (concat, length, ordering/equality, containment, formatting, identifier checks, partition/rpartition, translate).
Bug Fixes
- Improved decoding validation and edge-case handling (e.g., negative sizes and null input behavior) for more predictable failures.

coderabbitai · 2026-07-13T13:22:59Z

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

@coderabbitai resume to resume automatic reviews.
@coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

▶️ Resume reviews
🔍 Trigger review

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yml

Review profile: CHILL

Plan: Pro

Run ID: acd21ceb-a8ba-402f-acfe-0b13a4608a71

📥 Commits

Reviewing files that changed from the base of the PR and between acbcaaf and 37fd030.

📒 Files selected for processing (2)

crates/capi/src/unicodeobject.rs
crates/vm/src/builtins/str.rs

🚧 Files skipped from review as they are similar to previous changes (2)

crates/vm/src/builtins/str.rs
crates/capi/src/unicodeobject.rs

📝 Walkthrough

Walkthrough

The Unicode C API gains constructors, codec-based encoding and decoding helpers, comparisons, interning, containment, formatting, partitioning, translation, and identifier checks. Four PyStr methods become publicly callable for these APIs.

Changes

Unicode C API

Layer / File(s)	Summary
Unicode construction and encoding `crates/capi/src/unicodeobject.rs`	Adds string, object, and ordinal constructors, shared encoding support, and multiple Unicode encoding helpers.
Unicode decoding `crates/capi/src/unicodeobject.rs`	Adds general and encoding-specific decoding functions with size and pointer validation and codec-registry delegation.
Unicode operations and VM method access `crates/capi/src/unicodeobject.rs`, `crates/vm/src/builtins/str.rs`	Adds concatenation, length, interning, comparisons, equality, containment, formatting, partitioning, translation, and identifier checks while exposing the required `PyStr` methods.

Estimated code review effort: 4 (Complex) | ~45 minutes

Sequence Diagram(s)

sequenceDiagram
  participant CCaller
  participant UnicodeAPI
  participant PyStr
  participant codec_registry
  CCaller->>UnicodeAPI: provide Unicode object or encoded bytes
  UnicodeAPI->>PyStr: downcast and obtain string text
  UnicodeAPI->>codec_registry: encode_text or decode_text
  codec_registry-->>UnicodeAPI: encoded or decoded object
  UnicodeAPI-->>CCaller: return result

Possibly related PRs

RustPython/RustPython#7904: Extends the same Unicode C-API module with related constructors and encoding or comparison entrypoints.
RustPython/RustPython#8044: Adds related Unicode C-API encoding and decoding integration with the VM codec registry.

Suggested reviewers: youknowone

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title accurately summarizes the main change: adding more Unicode C-API functions.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

coderabbitai

Actionable comments posted: 2

🧹 Nitpick comments (1)

crates/capi/src/unicodeobject.rs (1)
98-171: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Collapse the six near-identical PyUnicode_As*String helpers onto a shared inner function.

Each of the six exported helpers has an identical body differing only by the encoding-name literal. Keep the individual #[no_mangle] symbols (required for the ABI) but delegate to one private helper to avoid drift.
♻️ Suggested shared helper
fn encode_as(unicode: *mut PyObject, encoding: &str) -> *mut PyObject {
    with_vm(|vm| {
        let unicode = unsafe { &*unicode }.try_downcast_ref::<PyStr>(vm)?.to_owned();
        vm.state.codec_registry.encode_text(unicode, encoding, None, vm)
    })
}

#[unsafe(no_mangle)]
pub unsafe extern "C" fn PyUnicode_AsASCIIString(unicode: *mut PyObject) -> *mut PyObject {
    encode_as(unicode, "ascii")
}
// ...and so on for latin-1, raw-unicode-escape, utf-16, utf-32, unicode-escape
As per coding guidelines: "When branches differ only in a value but share common logic, extract the differing value first, then call the common logic once to avoid duplicate code."
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@crates/capi/src/unicodeobject.rs` around lines 98 - 171, Extract the
duplicated with_vm/downcast/encode_text logic from PyUnicode_AsASCIIString,
PyUnicode_AsLatin1String, PyUnicode_AsRawUnicodeEscapeString,
PyUnicode_AsUTF16String, PyUnicode_AsUTF32String, and
PyUnicode_AsUnicodeEscapeString into one private encode_as helper accepting the
encoding name, then have each exported ABI function delegate to it with its
existing literal while preserving all #[unsafe(no_mangle)] symbols.
Source: Coding guidelines

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@crates/capi/src/unicodeobject.rs`:
- Around line 50-60: Update PyUnicode_FromObject to reject objects that are not
PyStr or PyStr subclasses instead of calling obj.to_owned().str(vm). Return the
existing C-API error representation for a TypeError, while preserving the
successful conversion path for accepted string objects.
- Around line 277-293: Update PyUnicode_CompareWithASCIIString to avoid calling
PyStr::expect_str(), which can panic for surrogate-containing strings. Compare
the PyStr value using its WTF-8 view, matching the behavior of
PyUnicode_Compare, while preserving the existing ordering-to-integer mapping and
UTF-8 validation for the C string.

---

Nitpick comments:
In `@crates/capi/src/unicodeobject.rs`:
- Around line 98-171: Extract the duplicated with_vm/downcast/encode_text logic
from PyUnicode_AsASCIIString, PyUnicode_AsLatin1String,
PyUnicode_AsRawUnicodeEscapeString, PyUnicode_AsUTF16String,
PyUnicode_AsUTF32String, and PyUnicode_AsUnicodeEscapeString into one private
encode_as helper accepting the encoding name, then have each exported ABI
function delegate to it with its existing literal while preserving all
#[unsafe(no_mangle)] symbols.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

Push a commit to this branch (recommended)
Create a new PR with the fixes

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yml

Review profile: CHILL

Plan: Pro

Run ID: a08d9e91-1068-4efd-8524-7620fca90951

📥 Commits

Reviewing files that changed from the base of the PR and between a9c2c52 and ecde4ab.

📒 Files selected for processing (1)

crates/capi/src/unicodeobject.rs

bschoenmaeckers · 2026-07-13T14:29:39Z

@coderabbitai review

coderabbitai · 2026-07-13T14:29:45Z

✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

coderabbitai

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@crates/capi/src/unicodeobject.rs`:
- Around line 578-591: Update PyUnicode_Translate to honor the _errors parameter
by converting and passing it through to PyStr::translate, preserving the
requested codec error-handler behavior; if PyStr::translate cannot support it,
explicitly document the intentional API divergence instead.
- Around line 258-331: Update PyUnicode_DecodeUTF16 and PyUnicode_DecodeUTF32 to
honor the _byteorder pointer: when it is non-NULL, select the corresponding
forced little-endian or big-endian codec based on its input value, while
preserving BOM/native-order decoding when it is NULL. After decoding, write the
detected byte order back through the pointer as required by the API.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

Push a commit to this branch (recommended)
Create a new PR with the fixes

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yml

Review profile: CHILL

Plan: Pro

Run ID: c01ec87c-2d24-4be3-b3ab-de829f116906

📥 Commits

Reviewing files that changed from the base of the PR and between 12667c2 and 1aa1771.

📒 Files selected for processing (2)

crates/capi/src/unicodeobject.rs
crates/vm/src/builtins/str.rs

bschoenmaeckers · 2026-07-17T17:09:42Z

@coderabbitai review

coderabbitai · 2026-07-17T17:10:10Z

✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

youknowone

👍

Review Add unicode decode functions Add more functions

youknowone

👍

bschoenmaeckers changed the title ~~Add more unicode functions~~ Add more unicode functions to the c-api Jul 13, 2026

coderabbitai Bot reviewed Jul 13, 2026

View reviewed changes

Comment thread crates/capi/src/unicodeobject.rs

Comment thread crates/capi/src/unicodeobject.rs

bschoenmaeckers marked this pull request as draft July 13, 2026 14:15

bschoenmaeckers force-pushed the c-api-simple-unicode branch from 12667c2 to 1aa1771 Compare July 14, 2026 15:07

bschoenmaeckers marked this pull request as ready for review July 14, 2026 15:07

coderabbitai Bot reviewed Jul 14, 2026

View reviewed changes

Comment thread crates/capi/src/unicodeobject.rs

Comment thread crates/capi/src/unicodeobject.rs

bschoenmaeckers marked this pull request as draft July 14, 2026 21:25

coderabbitai Bot mentioned this pull request Jul 16, 2026

Add codecs support to c-api #8267

Merged

bschoenmaeckers force-pushed the c-api-simple-unicode branch from 1aa1771 to 3a63323 Compare July 17, 2026 07:56

bschoenmaeckers marked this pull request as ready for review July 17, 2026 17:09

youknowone requested changes Jul 18, 2026

View reviewed changes

Comment thread crates/capi/src/unicodeobject.rs

bschoenmaeckers added 3 commits July 20, 2026 15:59

Add more unicode functions

ea6bd45

Review Add unicode decode functions Add more functions

Remove

8a845b8

Add helper

37fd030

bschoenmaeckers force-pushed the c-api-simple-unicode branch from acbcaaf to 37fd030 Compare July 20, 2026 14:06

youknowone approved these changes Jul 22, 2026

View reviewed changes

youknowone merged commit 72daf7c into RustPython:main Jul 22, 2026
26 checks passed

bschoenmaeckers deleted the c-api-simple-unicode branch July 22, 2026 11:48

Uh oh!

Conversation

bschoenmaeckers commented Jul 13, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary by CodeRabbit

Uh oh!

coderabbitai Bot commented Jul 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Reviews paused

Walkthrough

Changes

Sequence Diagram(s)

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

bschoenmaeckers commented Jul 13, 2026

Uh oh!

coderabbitai Bot commented Jul 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

bschoenmaeckers commented Jul 17, 2026

Uh oh!

coderabbitai Bot commented Jul 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

youknowone left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

youknowone left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

bschoenmaeckers commented Jul 13, 2026 •

edited by coderabbitai Bot

Loading

coderabbitai Bot commented Jul 13, 2026 •

edited

Loading

coderabbitai Bot commented Jul 13, 2026 •

edited

Loading

coderabbitai Bot commented Jul 17, 2026 •

edited

Loading