Skip to content

Match CPython Clinic layout for 56 raw text signatures - #9758

Draft
youknowdot wants to merge 5 commits into
RustPython:mainfrom
youknowdot:fix/clinic-signature-layout-20261110
Draft

youknowdot wants to merge 5 commits into
RustPython:mainfrom
youknowdot:fix/clinic-signature-layout-20261110

Conversation

@youknowdot

@youknowdot youknowdot commented Nov 8, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Match CPython 3.15 Argument Clinic's raw signature wrapping at compile time. Complete parameter chunks, including /, *, defaults, and trailing commas, wrap at 72 columns; continuation indentation and Unicode character counting follow Clinic. Exact fixtures cover the original 56 layout discrepancies.

The module-receiver implementation has been rebuilt following review:

  • Every module function owns its defining module in the single zelf receiver. module_object, with_module_object, BIND_MODULE, and the with_module attribute option are removed.
  • The native call carries a borrowed module in Callee, outside FuncArgs. Ordinary functions incur no receiver prepend, clone, or downcast. Bound methods retain positional receiver binding.
  • #[pyfunction] recognizes a leading PyRef<PyModule> or &Py<PyModule> syntactically, uses a distinct adapter kind, and omits that receiver from Python's arguments. Canonical spelling and alias limitations are documented; PyModuleRef is explicitly rejected.
  • _imp.exec_builtin and _imp.exec_dynamic declare a defining-module receiver before their Python-visible target module. C-API adapters consume the separate receiver without changing method calls.
  • The three binascii decoders use ordinary #[pyfunction] registration and obtain their full CPython documentation from rustpython-doc, not duplicated Rust docstrings.

Rebased independently onto main c07e0bc954; not stacked on another open PR. No new dependencies or changes to Lib/test.

Validation

  • Release build, normal commit hooks, and formatting pass.
  • Rust workspace tests: 1,561 passed, 21 ignored. C-API tests: 150 passed, 2 ignored. Strict workspace/C-API Clippy (-D warnings) passes.
  • test_binascii test_inspect test_builtin test_importlib: 1,994 tests, 132 skipped; all four modules pass.
  • Signature/binascii snippets pass on RustPython and actual CPython 3.15.2, including all four pytest wrapper cases.
  • Seven representative builtins/module functions have identical metadata and normal-call results on the baseline, changed build, and CPython 3.15.2. The invalid-call corpus is unchanged from baseline; existing CPython diagnostic differences are not claimed as fixed.
  • Full extra_tests pytest: 495 passed, 20 failed on both baseline and changed builds, with exactly the same failing cases. These cover sandbox IPC/socket/OS limitations, disabled SQLite, and existing standalone-CPython/test compatibility differences. Additional profiler tests likewise reproduce only the baseline forkserver permission failure.
  • math.sqrt tight loop, same optimized build settings, CPU-pinned alternating runs, 9 samples of 3 million calls per build: median 424.8 → 421.2 ns/call (-0.84%, within run-to-run variation). No increased call cost was observed.

Hosted CI: pending for 1ca28a0d49769ad6cd25cb777ec70fb19a92984b.

AI assistance: OpenAI Codex assisted with implementation, regression tests, review, and validation.

@coderabbitai

coderabbitai Bot commented Nov 8, 2026 •

Copy link
Copy Markdown
Contributor

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

  • Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true
Pre-merge checks: Passed 1 · Inconclusive 1

Failed Failed checks: 1 inconclusive

Check name Status Explanation Resolution
Title check Pending This assessment has not completed. The check will be retried on the next pre-merge check run.
Passed Passed checks: 1 passed

Check name Status Explanation
Description Check Passed Check skipped - CodeRabbit’s high-level summary is enabled.
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@youknowone youknowone left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The shared renderer change looks right. RustPython's derive-generated signatures are the counterpart of Clinic-generated ones, so applying Clinic's wrapping rule (chunks including the trailing comma, wrap at 72 columns, indent by name() in the one shared const renderer is the right place for it. It runs at compile time and only changes the raw __text_signature__ layout; __doc__ and inspect.signature are unaffected, as the PR states. Example with CPython 3.15.1:

binascii.a2b_base64.__text_signature__
before: '($module, data, /, *, strict_mode=<unrepresentable>, padded=True, alphabet=BASE64_ALPHABET, ignorechars=<unrepresentable>, canonical=False)'
after:  '($module, data, /, *, strict_mode=<unrepresentable>,\n           padded=True, alphabet=BASE64_ALPHABET,\n           ignorechars=<unrepresentable>, canonical=False)'

Minor: total_tokens comes from a second dry run of write_args over an empty buffer. It is correct, but the renderer now runs twice per signature, which is worth a short comment.

The binascii hunk shows a structural problem

The three ItemDoc::static_text("a2b_base*(...)") strings had to be wrapped by hand because they bypass the renderer. Every future renderer change will need the same manual edit, and nothing catches a miss. The information is also duplicated: A2bBase64Args already carries the full signature metadata (py_default = "BASE64_ALPHABET", etc.).

These are hand-built PyMethodDefs only because the functions take their defining module:

fn a2b_base64(module: PyRef<PyModule>, args: A2bBase64Args, vm: &VirtualMachine)

and #[pyfunction] has no way to receive that; no existing #[pyfunction] does. Every C module function receives PyObject *module, so the structural fix is for #[pyfunction] to recognize a leading module: &Py<PyModule> / PyRef<PyModule> parameter, pass the bound __self__, and leave it out of the signature, just as vm is left out. Then the manual definitions and the module_exec registration loop go away, the signatures are generated from A2bBase64Args, and this PR's wrapping applies automatically.

The other hand-written signature texts are the same kind of thing: __new__($type, *args, **kwargs) in vm/context.rs, __replace__($self, /, **fields) in _ast/python.rs, and __reduce__($self, /) in types/structseq.rs and stdlib/_thread.rs. They are closures or out-of-class PyMethodDefs that cannot go through the macros. They are short, so wrapping does not affect them today, but a const helper that builds an ItemDoc from typed signature metadata would remove them too. The static_text uses in crates/capi carry C-provided text and are fine.

Not blocking for this PR. Please note the binascii edit as a stopgap; module-parameter support in #[pyfunction] can follow separately (or land first, making the hunk unnecessary).

— commented by Claude

@youknowdot
youknowdot force-pushed the fix/clinic-signature-layout-20261110 branch 2 times, most recently from 3d7cb99 to 6b4d65a Compare November 11, 2026 02:45
Comment thread crates/stdlib/src/binascii.rs Outdated
canonical: OptionalArg<PyObjectRef>,
}

/// Decode a line of base64 data.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

don't add this. it must be retrived from rustpython-doc

@youknowone

Copy link
Copy Markdown
Member

@youknowdot Please replace the #[pyfunction(with_module)] option and the PyMethodFlags::BIND_MODULE flag in this PR with the design below. The Clinic layout part of this PR is unaffected.

Why

In CPython every module-level C function receives its module as the first C argument (m_self); there is no opt-in. That is why every module function's __text_signature__ starts with $module (math.sqrt, _imp.exec_builtin, binascii.a2b_base64 all report ($module, ...) and __self__ is the module). A per-function option plus a Rust-only method flag emulates that with extra metadata instead of making it the structure.

Design

  1. Every module function receives its module. Internally, a #[pyfunction] in a #[pymodule] always gets its defining module as the receiver.
  2. Transport it like vm, not through FuncArgs. Pass the module out of band, the way &VirtualMachine is passed beside FuncArgs today: either an extra parameter of the native call ABI (PyNativeFn is currently Fn(&VirtualMachine, FuncArgs, Callee)) or a field of Callee; your choice. Do not use args.prepend_arg(...) for module functions: functions that do not use the module must pay nothing on the hot path. Method receivers keep their current path; this change is for module functions only.
  3. Declare it by type, injected like vm. In the #[pyfunction] macro, if the first Rust parameter is PyRef<PyModule> or &Py<PyModule>, it is the receiver and is excluded from the Python signature. Otherwise the generated binding accepts the receiver and drops it. This must be matched syntactically in the macro: PyRef<PyModule> already implements the argument-conversion traits, so an IntoPyNativeFn impl keyed on it would overlap with the generic one. Decide how aliases such as PyModuleRef are handled (recognize them explicitly or reject them with a clear error) instead of silently treating them as a Python argument.
  4. A leading module parameter is always the receiver. _imp.exec_builtin and _imp.exec_dynamic (crates/vm/src/stdlib/_imp.rs) currently take the target module as their only parameter; rewrite them as (_imp: PyRef<PyModule>, mod: PyRef<PyModule>) so the first is the receiver and the second is the Python argument.
  5. Merge module_object into zelf. With a receiver always present, PyNativeFunction keeps a single zelf. Remove module_object and with_module_object, and update Py<PyModule>::__init_methods and the other users (function/method.rs, stdlib/builtins.rs, stdlib/_lsprof.rs, vm/compile.rs, builtins/builtin_func.rs). PyNativeFunction::call must then tell a module receiver (out of band) from a method receiver (prepended) without adding work for module functions.
  6. binascii. The decoders just declare module: PyRef<PyModule> as their first parameter; no option, no flag. Remove with_module from ALLOWED_NAMES and drop BIND_MODULE.

Must stay identical

  • __self__, __module__, __qualname__, repr(), __reduce__, ==/hash of module functions, including the _imp.exec_builtin(sys) and defining-module __self__ regression checks you already added.
  • __text_signature__ keeps the $module prefix; the receiver never appears as a Python parameter.
  • No new per-call cost for module functions that do not take the receiver.

Validation

Workspace and CAPI tests, strict Clippy for both, the existing signature/binascii snippets on CPython 3.15 and RustPython, and test_binascii test_inspect test_builtin test_importlib. Please also report a quick before/after micro-benchmark of a module function that does not take the module (for example math.sqrt in a tight loop).

— commented by Claude

Assisted-by: OpenAI Codex:GPT-6.1
Add explicit pyfunction(with_module) support for a leading PyRef<PyModule> or &Py<PyModule>, reusing the existing bound native call path. Keep this opt-in: _imp.exec_builtin and exec_dynamic take ordinary module arguments of the same type. Generate binascii decoder signatures from FromArgs metadata, preserve their docs, and remove manual registration.

Assisted-by: Codex:unknown
Remove the three inline docstring overrides from the module-bound decoder functions. Import their complete CPython 3.15.1 documentation into the shared database, including the missing base32 and base85 entries, and cover the parameter documentation in the existing binascii snippet.

Assisted-by: Codex:unknown
Replace the with_module opt-in and BIND_MODULE flag with a structural
module receiver carried in Callee. Infer leading owned or borrowed module
parameters in pyfunction through a distinct adapter, and keep one typed
receiver in PyNativeFunction while preserving positional method binding.

Update C-API adapters and _imp target-module calls, document the canonical
receiver spelling, and cover module injection and callable metadata. The
binascii decoders retain generated signatures and rustpython-doc bodies.

Assisted-by: Codex:unknown
@youknowdot
youknowdot force-pushed the fix/clinic-signature-layout-20261110 branch from 34ce070 to 1ca28a0 Compare November 12, 2026 16:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants