Skip to content

Commit fdd101b

Browse files
zzarbttooclaudeyouknowone
authored
Fix int unicode decimal digits (RustPython#8521)
* Accept Unicode decimal digits in int(), Decimal() and complex() CPython runs a string argument through _PyUnicode_TransformDecimalAndSpaceToASCII before parsing it, so decimal digits from any script are accepted: int('١٢٣') # 123 int('0x١f', 16) # 31 Decimal('١٢٣') # Decimal('123') complex('１+２j') # (1+2j) RustPython only did this for float(), which had the transform inlined. int() handed the raw UTF-8 bytes to bytes_to_int(), whose digit check is is_ascii_alphanumeric(), so every non-ASCII digit was rejected — even though float() accepted the same string. Lift the inlined transform out of float_from_string() into common::str::transform_decimal_and_space_to_ascii() and apply it to the str paths of int() and complex() too. The result is always ASCII: as in CPython, a character that is neither ASCII, whitespace nor a decimal digit becomes '?' and truncates the string, which no parser accepts at any base, leaving the caller to raise the error from the original string. Bytes-like input keeps going straight to the parser, matching CPython's split between PyLong_FromUnicodeObject and PyLong_FromString. This unmarks two expectedFailure tests: test_int.test_unicode and test_decimal.test_unicode_digits. * Share one PyStr-to-numeric-literal step across int, float and complex All three constructors need the same thing from a str argument: trim it, fold Unicode decimal digits and whitespace to ASCII, and give up on a string holding surrogates. Each expressed that last part differently — float matched PyKindStr and returned b"", complex leaned on to_str() returning None, int returned an empty Cow — so the rule lived in three places at once. Move it into protocol::numeric_literal_from_str() and have all three call it. CPython repeats this per type because its wrapper is three lines over a single PyUnicode representation; ours has to match over Ascii/Utf8/Wtf8, which is worth writing once. Only the shared step moves. int keeps its base handling, int and float keep accepting bytes-like input, complex keeps rejecting it, and each keeps raising its own error, because none of that is shared. No behavior change: the CPython differential suite is byte-identical before and after. * Drop the now-empty test_unicode_digits override in test_decimal Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com> Co-authored-by: Jeong, YunWon <69878+youknowone@users.noreply.github.com>
1 parent bf46487 commit fdd101b

8 files changed

Lines changed: 88 additions & 36 deletions

File tree

‎Lib/test/test_decimal.py‎

Lines changed: 0 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -854,10 +854,6 @@ class CExplicitConstructionTest(ExplicitConstructionTest, unittest.TestCase):
854854
class PyExplicitConstructionTest(ExplicitConstructionTest, unittest.TestCase):
855855
decimal = P
856856

857-
@unittest.expectedFailure # TODO: RUSTPYTHON
858-
def test_unicode_digits(self):
859-
return super().test_unicode_digits()
860-
861857
class ImplicitConstructionTest:
862858
'''Unit tests for Implicit Construction cases of Decimal.'''
863859

‎Lib/test/test_int.py‎

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -247,7 +247,6 @@ def test_invalid_signs(self):
247247
with self.assertRaises(ValueError):
248248
int(' + 1 ')
249249

250-
@unittest.expectedFailure # TODO: RUSTPYTHON
251250
def test_unicode(self):
252251
self.assertEqual(int("१२३४५६७८९०1234567890"), 12345678901234567890)
253252
self.assertEqual(int('١٢٣٤٥٦٧٨٩٠'), 1234567890)

‎crates/common/src/str.rs‎

Lines changed: 54 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,7 @@ use crate::atomic::{OncePtr, PyAtomic, Radium};
33
use crate::format::CharLen;
44
use crate::wtf8::{CodePoint, Wtf8, Wtf8Buf};
55
use crate::wtf8_index::Wtf8Index;
6+
use alloc::borrow::Cow;
67
use ascii::{AsciiChar, AsciiStr, AsciiString};
78
use core::fmt;
89
use core::ops::{Bound, RangeBounds};
@@ -835,10 +836,63 @@ pub fn char_to_decimal(ch: char) -> Option<u8> {
835836
.map(|i| (i % 10) as u8)
836837
}
837838

839+
/// Replace Unicode decimal digits with their ASCII equivalents and any Unicode
840+
/// whitespace with a plain space, so the byte-oriented numeric parsers can read
841+
/// them. Mirrors CPython's `_PyUnicode_TransformDecimalAndSpaceToASCII`.
842+
///
843+
/// The result is always ASCII. Any other non-ASCII character cannot appear in a
844+
/// numeric literal, so it becomes a `?` and the rest of the string is dropped:
845+
/// `?` is rejected by every parser at every base, which leaves the caller — the
846+
/// one that knows the base and owns the original string — to raise the error.
847+
#[must_use]
848+
pub fn transform_decimal_and_space_to_ascii(s: &str) -> Cow<'_, str> {
849+
if s.is_ascii() {
850+
return Cow::Borrowed(s);
851+
}
852+
let mut out = String::with_capacity(s.len());
853+
for c in s.chars() {
854+
if (c as u32) < 127 {
855+
out.push(c);
856+
} else if c.is_whitespace() {
857+
out.push(' ');
858+
} else if let Some(n) = char_to_decimal(c) {
859+
out.push(char::from_digit(n.into(), 10).unwrap());
860+
} else {
861+
out.push('?');
862+
break;
863+
}
864+
}
865+
debug_assert!(out.is_ascii());
866+
Cow::Owned(out)
867+
}
868+
838869
#[cfg(test)]
839870
mod tests {
840871
use super::*;
841872

873+
#[test]
874+
fn transform_decimal_and_space() {
875+
// ASCII input is passed through untouched, without allocating.
876+
assert!(matches!(
877+
transform_decimal_and_space_to_ascii("123"),
878+
Cow::Borrowed("123")
879+
));
880+
// Decimal digits from any script fold to ASCII.
881+
assert_eq!(transform_decimal_and_space_to_ascii("١٢٣"), "123");
882+
assert_eq!(transform_decimal_and_space_to_ascii("１２३"), "123");
883+
assert_eq!(transform_decimal_and_space_to_ascii("1٢3"), "123");
884+
// Unicode whitespace folds to a plain space.
885+
assert_eq!(transform_decimal_and_space_to_ascii("\u{3000}٣"), " 3");
886+
// ASCII characters ride through untouched, whatever they are.
887+
assert_eq!(transform_decimal_and_space_to_ascii("0x١f"), "0x1f");
888+
assert_eq!(transform_decimal_and_space_to_ascii("-١_٢"), "-1_2");
889+
// Anything else poisons the literal and truncates it, so the result stays
890+
// ASCII and the caller's parser is guaranteed to reject it.
891+
assert_eq!(transform_decimal_and_space_to_ascii("½가"), "?");
892+
assert_eq!(transform_decimal_and_space_to_ascii("١٢가٣"), "12?");
893+
assert_eq!(transform_decimal_and_space_to_ascii("١\u{7f}"), "1?");
894+
}
895+
842896
#[test]
843897
fn get_chars_basic() {
844898
let s = "0123456789";

‎crates/vm/src/builtins/complex.rs‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -220,10 +220,10 @@ impl Constructor for PyComplex {
220220
"complex() can't take second arg if first is a string",
221221
));
222222
}
223-
let (re, im) = s
224-
.to_str()
225-
.and_then(rustpython_literal::complex::parse_str)
226-
.ok_or_else(|| vm.new_value_error("complex() arg is a malformed string"))?;
223+
let (re, im) = rustpython_literal::complex::parse_str(
224+
&crate::protocol::numeric_literal_from_str(s),
225+
)
226+
.ok_or_else(|| vm.new_value_error("complex() arg is a malformed string"))?;
227227
return Ok(Self::from(Complex64 { re, im }));
228228
} else {
229229
return Err(vm.new_type_error(format!(

‎crates/vm/src/builtins/float.rs‎

Lines changed: 2 additions & 23 deletions
Original file line numberDiff line numberDiff line change
@@ -210,29 +210,8 @@ impl Constructor for PyFloat {
210210
pub fn float_from_string(val: PyObjectRef, vm: &VirtualMachine) -> PyResult<f64> {
211211
let (bytearray, buffer, buffer_lock, mapped_string);
212212
let b = if let Some(s) = val.downcast_ref::<PyStr>() {
213-
use crate::common::str::PyKindStr;
214-
match s.as_str_kind() {
215-
PyKindStr::Ascii(s) => s.trim().as_bytes(),
216-
PyKindStr::Utf8(s) => {
217-
mapped_string = s
218-
.trim()
219-
.chars()
220-
.map(|c| {
221-
if let Some(n) = rustpython_common::str::char_to_decimal(c) {
222-
char::from_digit(n.into(), 10).unwrap()
223-
} else if c.is_whitespace() {
224-
' '
225-
} else {
226-
c
227-
}
228-
})
229-
.collect::<String>();
230-
mapped_string.as_bytes()
231-
}
232-
// if there are surrogates, it's not gonna parse anyway,
233-
// so we can just choose a known bad value
234-
PyKindStr::Wtf8(_) => b"",
235-
}
213+
mapped_string = crate::protocol::numeric_literal_from_str(s);
214+
mapped_string.as_bytes()
236215
} else if let Some(bytes) = val.downcast_ref::<PyBytes>() {
237216
bytes.as_bytes()
238217
} else if let Some(buf) = val.downcast_ref::<PyByteArray>() {

‎crates/vm/src/builtins/int.rs‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -16,7 +16,7 @@ use crate::{
1616
ArgByteOrder, ArgIntoBool, FuncArgs, OptionalArg, OptionalOption, PyArithmeticValue,
1717
PyComparisonValue,
1818
},
19-
protocol::{PyNumberMethods, handle_bytes_to_int_err},
19+
protocol::{PyNumberMethods, handle_bytes_to_int_err, numeric_literal_from_str},
2020
types::{AsNumber, Comparable, Constructor, Hashable, PyComparisonOp, Representable},
2121
};
2222
use alloc::fmt;
@@ -822,7 +822,7 @@ struct IntToByteArgs {
822822
fn try_int_radix(obj: &PyObject, base: u32, vm: &VirtualMachine) -> PyResult<BigInt> {
823823
match_class!(match obj.to_owned() {
824824
string @ PyStr => {
825-
let s = string.as_wtf8().trim();
825+
let s = numeric_literal_from_str(&string);
826826
bytes_to_int(s.as_bytes(), base, vm.state.int_max_str_digits.load())
827827
.map_err(|e| handle_bytes_to_int_err(e, obj, vm))
828828
}

‎crates/vm/src/protocol/mod.rs‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -16,5 +16,6 @@ pub use mapping::{PyMapping, PyMappingMethods, PyMappingSlots};
1616
pub use number::{
1717
PyNumber, PyNumberBinaryFunc, PyNumberBinaryOp, PyNumberMethods, PyNumberSlots,
1818
PyNumberTernaryFunc, PyNumberTernaryOp, PyNumberUnaryFunc, handle_bytes_to_int_err,
19+
numeric_literal_from_str,
1920
};
2021
pub use sequence::{PySequence, PySequenceMethods, PySequenceSlots};

‎crates/vm/src/protocol/number.rs‎

Lines changed: 25 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -8,11 +8,34 @@ use crate::{
88
builtins::{
99
PyBaseExceptionRef, PyByteArray, PyBytes, PyComplex, PyFloat, PyInt, PyIntRef, PyStr, int,
1010
},
11-
common::int::{BytesToIntError, bytes_to_int},
11+
common::{
12+
int::{BytesToIntError, bytes_to_int},
13+
str::{PyKindStr, transform_decimal_and_space_to_ascii},
14+
},
1215
function::ArgBytesLike,
1316
object::{Traverse, TraverseFn},
1417
stdlib::_warnings,
1518
};
19+
use alloc::borrow::Cow;
20+
21+
/// Normalize a `str` for the byte-oriented numeric parsers: Unicode decimal digits
22+
/// and whitespace fold to their ASCII equivalents, the way CPython runs every
23+
/// numeric constructor's string argument through
24+
/// `_PyUnicode_TransformDecimalAndSpaceToASCII` first.
25+
///
26+
/// `int`, `float` and `complex` share this step and nothing else — only `int` takes
27+
/// a base, and only `int` and `float` accept bytes-like input, so each keeps its own
28+
/// entry point around this one.
29+
///
30+
/// A string holding surrogates can never be a valid literal, so it folds to an
31+
/// empty — and therefore invalid — one.
32+
pub fn numeric_literal_from_str(s: &PyStr) -> Cow<'_, str> {
33+
match s.as_str_kind() {
34+
PyKindStr::Ascii(s) => Cow::Borrowed(s.trim().as_str()),
35+
PyKindStr::Utf8(s) => transform_decimal_and_space_to_ascii(s.trim()),
36+
PyKindStr::Wtf8(_) => Cow::Borrowed(""),
37+
}
38+
}
1639

1740
pub type PyNumberUnaryFunc<R = PyObjectRef> = fn(PyNumber<'_>, &VirtualMachine) -> PyResult<R>;
1841
pub type PyNumberBinaryFunc = fn(&PyObject, &PyObject, &VirtualMachine) -> PyResult;
@@ -59,7 +82,7 @@ impl PyObject {
5982
} else if let Some(i) = self.number().int(vm).or_else(|| self.try_index_opt(vm)) {
6083
i
6184
} else if let Some(s) = self.downcast_ref::<PyStr>() {
62-
try_convert(self, s.as_wtf8().trim().as_bytes(), vm)
85+
try_convert(self, numeric_literal_from_str(s).as_bytes(), vm)
6386
} else if let Some(bytes) = self.downcast_ref::<PyBytes>() {
6487
try_convert(self, bytes, vm)
6588
} else if let Some(bytearray) = self.downcast_ref::<PyByteArray>() {

0 commit comments

Comments
 (0)