Repository navigation
Description
RawJSON corrupts valid non-ASCII JSON with default ensure_ascii=True
On current master 0f1455d, valid Unicode RawJSON can become invalid JSON or raise UnicodeDecodeError under the default encoder settings:
import rapidjson as rj
rj.dumps(rj.RawJSON('"☃"')) # produces '"\x03"', not the original value
rj.dumps(rj.RawJSON('{"key": "é"}')) # UnicodeDecodeError
rj.dumps(rj.RawJSON('"☃"'), ensure_ascii=False) # '"☃"'The same issue occurs with Encoder, dump to StringIO/BytesIO, compact and pretty output. Expected: valid preserialized JSON should retain its semantic value. Given dumps' documented ensure_ascii guarantee, the proposed fix escapes non-ASCII code points as JSON Unicode escapes when ensure_ascii=True; False retains the literal content. ASCII spelling and whitespace are unchanged. RawJSON still does not validate arbitrary input.
Cause: RapidJSON RawValue transcodes Unicode into the ASCII target without JSON escaping, truncating code points to bytes. This differs from the performance discussion in #169; no performance speedup is claimed here. I inspected that discussion, all five currently open PRs, RawJSON test history, recent rapidjson.cpp commits, and targeted all-state searches without finding a Unicode-corruption fix.
Linux CPython 3.14.8, locally rebuilt extension. 24 new regression cases fail on rebuilt unchanged source; patched full tests, doctests and stubtest results are in the paired PR. This is a local reproduction, not a claimed production incident.
Compatibility decision: the existing RawJSON docs promise verbatim insertion including whitespace, while dumps docs say all incoming non-ASCII characters are escaped under ensure_ascii=True. Current behavior satisfies neither for these inputs: it truncates Unicode or raises rather than inserting the original value. The proposal gives the global ASCII setting precedence for valid RawJSON string content. That deliberately changes literal non-ASCII spelling to escapes under the default; callers who require exact Unicode text must use ensure_ascii=False. An alternative would preserve Unicode verbatim regardless of ensure_ascii, but would weaken that documented global guarantee. ASCII content, existing escapes and whitespace are not normalized, and arbitrary raw content is still not validated. Maintainer preference on this compatibility tradeoff is welcome.
Activity
lelit commentedon Nov 3, 2026
It's a real tension and I do not have a good answer.
On one hand,
RawJSONwere introduced (PR #95) to handle "some preserialized JSON", and thus it was intentionally built to be some kind of "black box". OTOH, I think PyRJ should try hard to avoid a) hard errors and b) wrong/unparseable output.So, I think that your PR is minimal and reasonable. Any other thing would require a parse/reserialize dance on
RawJSONpayloads, that would defeat its purpose.