exec(), eval() and -c keep a lone CR in string literals when the source is a str #9761

Summary

When source code is passed as a str to exec() or eval(), or with rustpython -c, a lone carriage return inside a string literal stays '\r'. CPython reads it as '\n', as it does for CRLF. So code saved with CR-only line endings gets different string values and docstrings in RustPython. Docstrings are also left uncleaned, because a docstring with only CR line breaks counts as one line. Within RustPython, the same text also gives different values depending on the call: exec(text) keeps the CR, but exec(text.encode()) and exec(compile(text, ...)) turn it into '\n'.

The language reference ("Physical lines") says: "Regardless of platform, each of these sequences is replaced by a single ASCII LF (linefeed) character. (This is done even inside string literals.)"

Reproducer

# source saved with old Mac (CR-only) line endings, e.g. read from a zip or with pkgutil.get_data()
data = b'"""Usage:\r  tool run"""\rBANNER = """line one\rline two"""\r'
text = data.decode()

ns = {}; exec(data, ns)
print("exec(bytes)        ", ascii(ns["__doc__"]), ascii(ns["BANNER"]))
ns = {}; exec(compile(text, "<s>", "exec"), ns)
print("exec(compile(str)) ", ascii(ns["__doc__"]), ascii(ns["BANNER"]))
ns = {}; exec(text, ns)
print("exec(str)          ", ascii(ns["__doc__"]), ascii(ns["BANNER"]))
print("eval(str)          ", ascii(eval('"""a\rb"""')))

Expected

CPython 3.15.2 (3.13.18 prints the same; 3.12.13 differs only in not dedenting the docstring):

exec(bytes)         'Usage:\ntool run' 'line one\nline two'
exec(compile(str))  'Usage:\ntool run' 'line one\nline two'
exec(str)           'Usage:\ntool run' 'line one\nline two'
eval(str)           'a\nb'

Actual

RustPython main (b66534d), same on ef023bc:

exec(bytes)         'Usage:\ntool run' 'line one\nline two'
exec(compile(str))  'Usage:\ntool run' 'line one\nline two'
exec(str)           'Usage:\r  tool run' 'line one\rline two'
eval(str)           'a\rb'

rustpython -c $'x = """a\rb"""\nprint(ascii(x))' prints 'a\rb' (CPython 'a\nb').

On the exec(str) / eval(str) / -c paths, every literal kind keeps the CR: plain, raw (r"""a<CR>b"""), bytes (b"""...""", rb"""..."""), f-strings and t-strings (literal parts and a format spec), and a raw string with a backslash before the CR (r"a\<CR>b" gives 'a\\\rb', CPython 'a\\\nb'). CRLF is handled correctly on these paths. Everything else is already correct: exec/eval of bytes, compile() of str or bytes, ast.parse, codeop, importing a CR-only module, running a CR-only script file, and stdin. Statements, comments and line numbers already treat a lone CR as a line break, for both str and bytes source.

Cause

#9429 added newline normalization (\r\n and \r to \n) to the source decoder (compile.rs#L464-L470). compile() always goes through that decoder, and so do exec/eval of bytes. Its description already notes that "direct string paths that bypass the decoder are not changed". exec() and eval() with a str pass the text straight to run_code (builtins.rs#L640-L650, #L591-L601, #L701-L708), and -c goes through vm.run_string. Both end in compile_with_syntax_warning_handler, which only replaces "\r\n" (lib.rs#L2409-L2421). The parser copies literal text from the source as is (string.rs#L499-L518 in the ruff fork; upstream ruff tracks the same parser behavior as astral-sh/ruff#16318). So any CR that reaches it unnormalized ends up in the value.

I tried changing line 2416 to source.replace("\r\n", "\n").replace('\r', "\n") locally. With that change, all the cases above match CPython 3.15. Running the str paths through the same normalization as the decoder would work too.

CPython's own tests do not catch this: test_source_encoding's test_crlf/test_crcrlf cases go through bytes and script files, and test_compile.test_other_newlines uses compile(). All of these pass.

_{Drafted with help from Claude Code; I reviewed and checked it before posting.}