Skip to content

Commit 322f294

Browse files
youknowoneyouknowone
authored
common: add VM-independent codec and text engines (#8622)
* common: add PyPy CJK codec engines Assisted-by: Codex:GPT-5 * common: mark CJK mapping tables generated Assisted-by: Codex:GPT-5 * common: add reusable byte and text engines Assisted-by: Codex:GPT-5 * common: record opt-in engine dependencies Assisted-by: Codex:GPT-5 * stdlib: add `_multibytecodec` and the six `_codecs_*` modules Wire the CJK codec engines in `rustpython-common` up to the VM. `_multibytecodec` holds `MultibyteCodec`, `MultibyteIncrementalEncoder`, `MultibyteIncrementalDecoder`, `MultibyteStreamReader` and `MultibyteStreamWriter`, the encode/decode drivers and the error-handler protocol; each `_codecs_*` module exposes `getcodec` for the encodings of its region. `rustpython-stdlib` now enables the `cjk-codecs` feature of `rustpython-common`. Add `cjk::decode_reset`, which the decoder `reset()` methods need. It differs from building a fresh decoder state: ISO 2022 keeps its G1..G3 designations and the escape-throughout flag, and resets only G0 and the shifted flag. Drop the markers from the CJK codec tests and from the `test_codecs`, `test_io`, `test_email`, `test_zipfile` and `test_traceback` cases that needed a CJK codec. `test_codecs.BasicUnicodeTest.{test_basics,test_decoder_state}` stay expected failures for an unrelated reason: `codecs.charmap_encode` rejects a `None` mapping. Assisted-by: Claude Code:claude-opus-5 * stdlib: use the byte and text engines from `rustpython-common` `binascii`, `_json` and `_socket`'s address converters now call the engines in `rustpython-common` instead of carrying their own copies, and `rustpython-stdlib` enables the `binascii`, `inet` and `json` features for them. That leaves `base64` and `crc32fast` with no caller in `rustpython-stdlib`, and `crates/stdlib/src/json/machinery.rs`, the last of the json_in_type-derived code, goes away with its last caller. The address converters change behaviour. `inet_aton` reads the lenient forms (`127.1`, `16909060`, `0x7f.1`, `0177.0.0.1`), `inet_pton` rejects an octet with a redundant leading zero and a group of five hex digits, `inet_ntop` writes the IPv4-compatible form as `::c000:201`, and an address family with no converter raises `EAFNOSUPPORT` instead of a bare `OSError`. Give the JSON engine the closing-quote scan and the allocation-free `\uXXXX` escape it needs to carry `_json`'s workload. Against the code it replaces, `json.dumps` runs 2.3x faster with `ensure_ascii=True` and 1.7x without it, and `json.loads` 1.2x. `binascii.a2b_base64` now reports the messages `binascii.c` does: no `error decoding base64: ` prefix, and the data-character count in parentheses. Assisted-by: Claude Code:claude-opus-5 * test: enable the Windows cp932 codec case The shared CJK engine now satisfies the Windows code-page adapter, so keeping this test marked as an expected failure turns its success into a CI failure. Assisted-by: Codex:GPT-5
1 parent d1bbf4b commit 322f294

50 files changed

Lines changed: 47279 additions & 1258 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.cspell.dict/cpython.txt‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -37,6 +37,7 @@ cfws
3737
CFWS
3838
chans
3939
cidx
40+
cjkcodecs
4041
CLASSDEREF
4142
classdict
4243
cmpop
@@ -65,6 +66,7 @@ elts
6566
eooh
6667
eofs
6768
EOOH
69+
esize
6870
evalloop
6971
excepthandler
7072
exceptiontable
@@ -138,6 +140,9 @@ lslpp
138140
lsprof
139141
MAXBLOCKS
140142
maxdepth
143+
mbiencoder
144+
mbstreamreader
145+
mbstreamwriter
141146
metavars
142147
miscompiles
143148
mult

‎.cspell.json‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -52,6 +52,22 @@
5252
"Lib/**",
5353
"crates/host_env/**"
5454
],
55+
"overrides": [
56+
{
57+
"filename": "crates/common/src/encodings/cjk/**",
58+
"words": [
59+
"CHOSEONG",
60+
"EUCKR",
61+
"famille",
62+
"GBCOMMON",
63+
"jong",
64+
"JONGSEONG",
65+
"JUNGSEONG",
66+
"PHINT",
67+
"Xabc"
68+
]
69+
}
70+
],
5571
// words - list of words to be always considered correct
5672
// (compound words like pyarg, baseclass, microbenchmark are handled by allowCompoundWords)
5773
"words": [

‎.gitattributes‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,7 @@ Lib/*.py text working-tree-encoding=UTF-8 eol=LF
66
crates/vm/src/stdlib/ast/gen.rs linguist-generated -merge
77
crates/doc/src/data.inc.rs generated
88
crates/compiler-core/src/bytecode/opcode_metadata.rs generated
9+
crates/common/src/encodings/cjk/mappings_*.rs generated
910

1011
.github/workflows/*.lock.yml linguist-generated=true merge=ours
1112

‎Cargo.lock‎

Lines changed: 3 additions & 2 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎Lib/test/test_codecencodings_cn.py‎

Lines changed: 0 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,6 @@
66
from test import multibytecodec_support
77
import unittest
88

9-
@unittest.skip("TODO: RUSTPYTHON; unknown encoding: gb2312")
109
class Test_GB2312(multibytecodec_support.TestBase, unittest.TestCase):
1110
encoding = 'gb2312'
1211
tstring = multibytecodec_support.load_teststring('gb2312')
@@ -20,7 +19,6 @@ class Test_GB2312(multibytecodec_support.TestBase, unittest.TestCase):
2019
(b"\xc1\x64", "strict", None),
2120
)
2221

23-
@unittest.skip("TODO: RUSTPYTHON; unknown encoding: gbk")
2422
class Test_GBK(multibytecodec_support.TestBase, unittest.TestCase):
2523
encoding = 'gbk'
2624
tstring = multibytecodec_support.load_teststring('gbk')
@@ -35,7 +33,6 @@ class Test_GBK(multibytecodec_support.TestBase, unittest.TestCase):
3533
("\u30fb", "strict", None),
3634
)
3735

38-
@unittest.skip("TODO: RUSTPYTHON; unknown encoding: gb18030")
3936
class Test_GB18030(multibytecodec_support.TestBase, unittest.TestCase):
4037
encoding = 'gb18030'
4138
tstring = multibytecodec_support.load_teststring('gb18030')
@@ -60,7 +57,6 @@ class Test_GB18030(multibytecodec_support.TestBase, unittest.TestCase):
6057
)
6158
has_iso10646 = True
6259

63-
@unittest.skip("TODO: RUSTPYTHON; unknown encoding: hz")
6460
class Test_HZ(multibytecodec_support.TestBase, unittest.TestCase):
6561
encoding = 'hz'
6662
tstring = multibytecodec_support.load_teststring('hz')

‎Lib/test/test_codecencodings_hk.py‎

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,6 @@
66
from test import multibytecodec_support
77
import unittest
88

9-
@unittest.skip("TODO: RUSTPYTHON; unknown encoding: big5hkscs")
109
class Test_Big5HKSCS(multibytecodec_support.TestBase, unittest.TestCase):
1110
encoding = 'big5hkscs'
1211
tstring = multibytecodec_support.load_teststring('big5hkscs')

‎Lib/test/test_codecencodings_iso2022.py‎

Lines changed: 0 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -10,23 +10,20 @@
1010
(b'ab\x1B$def', 'replace', 'ab\uFFFD'),
1111
)
1212

13-
@unittest.skip("TODO: RUSTPYTHON; unknown encoding: iso2022_jp")
1413
class Test_ISO2022_JP(multibytecodec_support.TestBase, unittest.TestCase):
1514
encoding = 'iso2022_jp'
1615
tstring = multibytecodec_support.load_teststring('iso2022_jp')
1716
codectests = COMMON_CODEC_TESTS + (
1817
(b'ab\x1BNdef', 'replace', 'ab\x1BNdef'),
1918
)
2019

21-
@unittest.skip("TODO: RUSTPYTHON; unknown encoding: iso2022_jp_2")
2220
class Test_ISO2022_JP2(multibytecodec_support.TestBase, unittest.TestCase):
2321
encoding = 'iso2022_jp_2'
2422
tstring = multibytecodec_support.load_teststring('iso2022_jp')
2523
codectests = COMMON_CODEC_TESTS + (
2624
(b'ab\x1BNdef', 'replace', 'abdef'),
2725
)
2826

29-
@unittest.skip("TODO: RUSTPYTHON; unknown encoding: iso2022_jp_3")
3027
class Test_ISO2022_JP3(multibytecodec_support.TestBase, unittest.TestCase):
3128
encoding = 'iso2022_jp_3'
3229
tstring = multibytecodec_support.load_teststring('iso2022_jp')
@@ -50,7 +47,6 @@ class Test_ISO2022_JP3(multibytecodec_support.TestBase, unittest.TestCase):
5047
b'\x1B$(O\x29\x28\x1B(Bℜ\x1B$(O\x29\x32\x1B(B = ⟨ሴ⟩'
5148
)
5249

53-
@unittest.skip("TODO: RUSTPYTHON; unknown encoding: iso2022_jp_2004")
5450
class Test_ISO2022_JP2004(multibytecodec_support.TestBase, unittest.TestCase):
5551
encoding = 'iso2022_jp_2004'
5652
tstring = multibytecodec_support.load_teststring('iso2022_jp')
@@ -74,7 +70,6 @@ class Test_ISO2022_JP2004(multibytecodec_support.TestBase, unittest.TestCase):
7470
b'\x1B$(Q\x29\x28\x1B(Bℜ\x1B$(Q\x29\x32\x1B(B = ⟨ሴ⟩'
7571
)
7672

77-
@unittest.skip("TODO: RUSTPYTHON; unknown encoding: iso2022_kr")
7873
class Test_ISO2022_KR(multibytecodec_support.TestBase, unittest.TestCase):
7974
encoding = 'iso2022_kr'
8075
tstring = multibytecodec_support.load_teststring('iso2022_kr')

‎Lib/test/test_codecencodings_jp.py‎

Lines changed: 0 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,6 @@
66
from test import multibytecodec_support
77
import unittest
88

9-
@unittest.skip("TODO: RUSTPYTHON; unknown encoding: cp932")
109
class Test_CP932(multibytecodec_support.TestBase, unittest.TestCase):
1110
encoding = 'cp932'
1211
tstring = multibytecodec_support.load_teststring('shift_jis')
@@ -40,7 +39,6 @@ class Test_CP932(multibytecodec_support.TestBase, unittest.TestCase):
4039
(b"\x8eXY", "replace", "\ufffdXY"),
4140
)
4241

43-
@unittest.skip("TODO: RUSTPYTHON; unknown encoding: euc_jis_2004")
4442
class Test_EUC_JIS_2004(multibytecodec_support.TestBase,
4543
unittest.TestCase):
4644
encoding = 'euc_jis_2004'
@@ -51,7 +49,6 @@ class Test_EUC_JIS_2004(multibytecodec_support.TestBase,
5149
b"\xa9\xa8ℜ\xa9\xb2 = ⟨ሴ⟩"
5250
)
5351

54-
@unittest.skip("TODO: RUSTPYTHON; unknown encoding: euc_jisx0213")
5552
class Test_EUC_JISX0213(multibytecodec_support.TestBase,
5653
unittest.TestCase):
5754
encoding = 'euc_jisx0213'
@@ -62,7 +59,6 @@ class Test_EUC_JISX0213(multibytecodec_support.TestBase,
6259
b"\xa9\xa8ℜ\xa9\xb2 = ⟨ሴ⟩"
6360
)
6461

65-
@unittest.skip("TODO: RUSTPYTHON; unknown encoding: euc_jp")
6662
class Test_EUC_JP_COMPAT(multibytecodec_support.TestBase,
6763
unittest.TestCase):
6864
encoding = 'euc_jp'
@@ -78,7 +74,6 @@ class Test_EUC_JP_COMPAT(multibytecodec_support.TestBase,
7874
(b"abc\x80\x80\x82\x84def", "ignore", "abc\uff44def"),
7975
)
8076

81-
@unittest.skip("TODO: RUSTPYTHON; unknown encoding: shift_jis")
8277
class Test_SJIS_COMPAT(multibytecodec_support.TestBase, unittest.TestCase):
8378
encoding = 'shift_jis'
8479
tstring = multibytecodec_support.load_teststring('shift_jis')
@@ -93,7 +88,6 @@ class Test_SJIS_COMPAT(multibytecodec_support.TestBase, unittest.TestCase):
9388
(b"abc\xFF\x58", "replace", "abc\ufffdX"),
9489
)
9590

96-
@unittest.skip("TODO: RUSTPYTHON; unknown encoding: shift_jis_2004")
9791
class Test_SJIS_2004(multibytecodec_support.TestBase, unittest.TestCase):
9892
encoding = 'shift_jis_2004'
9993
tstring = multibytecodec_support.load_teststring('shift_jis')
@@ -112,7 +106,6 @@ class Test_SJIS_2004(multibytecodec_support.TestBase, unittest.TestCase):
112106
b"\x85Gℜ\x85Q = ⟨ሴ⟩"
113107
)
114108

115-
@unittest.skip("TODO: RUSTPYTHON; unknown encoding: shift_jisx0213")
116109
class Test_SJISX0213(multibytecodec_support.TestBase, unittest.TestCase):
117110
encoding = 'shift_jisx0213'
118111
tstring = multibytecodec_support.load_teststring('shift_jisx0213')

‎Lib/test/test_codecencodings_kr.py‎

Lines changed: 0 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,6 @@
66
from test import multibytecodec_support
77
import unittest
88

9-
@unittest.skip("TODO: RUSTPYTHON; unknown encoding: cp949")
109
class Test_CP949(multibytecodec_support.TestBase, unittest.TestCase):
1110
encoding = 'cp949'
1211
tstring = multibytecodec_support.load_teststring('cp949')
@@ -19,7 +18,6 @@ class Test_CP949(multibytecodec_support.TestBase, unittest.TestCase):
1918
(b"abc\x80\x80\xc1\xc4", "ignore", "abc\uc894"),
2019
)
2120

22-
@unittest.skip("TODO: RUSTPYTHON; unknown encoding: euc_kr")
2321
class Test_EUCKR(multibytecodec_support.TestBase, unittest.TestCase):
2422
encoding = 'euc_kr'
2523
tstring = multibytecodec_support.load_teststring('euc_kr')
@@ -51,7 +49,6 @@ class Test_EUCKR(multibytecodec_support.TestBase, unittest.TestCase):
5149
(b"\xc1\xc4", "strict", "\uc894"),
5250
)
5351

54-
@unittest.skip("TODO: RUSTPYTHON; unknown encoding: johab")
5552
class Test_JOHAB(multibytecodec_support.TestBase, unittest.TestCase):
5653
encoding = 'johab'
5754
tstring = multibytecodec_support.load_teststring('johab')

‎Lib/test/test_codecencodings_tw.py‎

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,6 @@
66
from test import multibytecodec_support
77
import unittest
88

9-
@unittest.skip("TODO: RUSTPYTHON; unknown encoding: big5")
109
class Test_Big5(multibytecodec_support.TestBase, unittest.TestCase):
1110
encoding = 'big5'
1211
tstring = multibytecodec_support.load_teststring('big5')

0 commit comments

Comments
 (0)