Summary

VirtualMachine::new_unicode_decode_error (and its sibling new_unicode_encode_error) construct a UnicodeDecodeError that is structurally invalid: it has none of the five attributes that a UnicodeDecodeError must have (encoding, object, start, end, reason), and its args is a 1‑tuple holding a plain message string. A correct helper — new_unicode_decode_error_real — already exists, and both macro-generated helpers are already marked for removal:

// crates/vm/src/vm/vm_new.rs
// TODO: remove & replace with new_unicode_decode_error_real
define_exception_fn!(fn new_unicode_decode_error, unicode_decode_error, UnicodeDecodeError);

// TODO: remove & replace with new_unicode_encode_error_real
define_exception_fn!(fn new_unicode_encode_error, unicode_encode_error, UnicodeEncodeError);

This issue tracks replacing the remaining call sites of the broken helpers with the _real variants and removing the broken helpers.

Why the macro version is wrong

new_unicode_decode_error is generated by the define_exception_fn! macro, which just calls new_exception_msg:

pub fn $fn_name(&self, msg: impl Into<Wtf8Buf>) -> PyBaseExceptionRef {
    let err = self.ctx.exceptions.$attr.to_owned();
    self.new_exception_msg(err, msg.into())   // -> new_exception(type, vec![msg_str])
}

new_exception builds the exception via PyBaseException::new(args, vm), which only stores args. It does not run slot_init, so PyUnicodeDecodeError's initializer — the only thing that sets encoding/object/start/end/reason — never runs. The resulting object is a UnicodeDecodeError with args = ("some message",) and no unicode-error attributes at all.

new_unicode_decode_error_real instead passes the full 5-tuple and sets every attribute, matching what UnicodeDecodeError(encoding, object, start, end, reason) produces.

Note that in CPython this object cannot even be constructed — UnicodeDecodeError requires exactly 5 arguments:

>>> UnicodeDecodeError('csv not utf8')
TypeError: function takes exactly 5 arguments (1 given)

Reproduction

csv.reader(..., quoting=csv.QUOTE_STRINGS) reaches crates/stdlib/src/csv.rs:1113, which uses the broken helper:

import csv
r = csv.reader(['\udcff'], quoting=csv.QUOTE_STRINGS)
try:
    next(r)
except UnicodeDecodeError as e:
    print("str(e) =", repr(str(e)))
    print("args   =", e.args)
    for attr in ("encoding", "object", "start", "end", "reason"):
        try:
            print(" ", attr, "=", repr(getattr(e, attr)))
        except AttributeError:
            print(" ", attr, "-> AttributeError")

RustPython output:

str(e) = ''
args   = ('csv not utf8',)
  encoding -> AttributeError
  object -> AttributeError
  start -> AttributeError
  end -> AttributeError
  reason -> AttributeError

Two concrete symptoms:

1. The diagnostic message is silently lost. UnicodeDecodeError.__str__ returns an empty string when the object attribute is missing, so str(e) is '' instead of the message that was passed in.
2. Attribute access raises AttributeError. Any error handler that inspects e.encoding / e.start / e.reason (as the codecs error-handler machinery and normal user code do) breaks. A genuine UnicodeDecodeError always has these attributes.

For comparison, a well-formed instance (CPython, and RustPython via new_unicode_decode_error_real):

str(e) = "'utf-8' codec can't decode byte 0xff in position 0: invalid start byte"
args   = ('utf-8', b'\xff', 0, 1, 'invalid start byte')
encoding = utf-8 | object = b'\xff' | start = 0 | end = 1 | reason = 'invalid start byte'

Call sites to convert

Correct (already using _real) — for reference:

• crates/vm/src/codecs.rs:805
• crates/vm/src/stdlib/_codecs.rs:1023, :1114, :1125
• crates/capi/src/pyerrors.rs:353

Broken — new_unicode_decode_error(msg):

• crates/stdlib/src/csv.rs:1113, :1239, :1415, :1461, :1510, :1584
• crates/vm/src/stdlib/nt.rs:218, :722, :724, :726, :938 (Windows)
• crates/stdlib/src/socket.rs:2616, :2637
• crates/vm/src/stdlib/_codecs.rs:483, :499, :608, :624 (mbcs/oem, Windows)
• crates/vm/src/stdlib/posix.rs:1281, :1737
• crates/vm/src/function/fspath.rs:129
• crates/vm/src/stdlib/os.rs:134

Broken — new_unicode_encode_error(msg) (same defect, UnicodeEncodeError side; _real variant is new_unicode_encode_error_real):

• crates/stdlib/src/array.rs:641, :1700
• crates/stdlib/src/socket.rs:2630
• crates/vm/src/stdlib/_codecs.rs:397, :431, :521, :555 (mbcs/oem, Windows)

Plan

1. At each broken decode call site, supply the real (encoding, object, start, end, reason) values and call new_unicode_decode_error_real. Most of these already hold the source bytes and the failing offset (or can compute them from Utf8Error::valid_up_to); where only a generic reason is available, pass "utf-8" as encoding, the original bytes as object, and the reason string. Do the same for the encode call sites with new_unicode_encode_error_real and (encoding, object: str, start, end, reason).
2. Remove new_unicode_decode_error and new_unicode_encode_error once their last callers are gone.

Related

• Several built-in exceptions store their attributes in the instance dict instead of as fields #8345 — separate but adjacent: even the correctly constructed UnicodeDecodeError leaks its five attributes into __dict__ instead of storing them as struct fields.

Investigated and drafted by Claude; reviewed before filing.