Summary
The C API documents that the buffers returned by PyBytes_AsString(), PyByteArray_AsString(), and PyUnicode_AsUTF8AndSize() to contain an extra trailing NUL byte. RustPython instead returns pointers to the objects' exact-length Rust buffers. Reading the documented byte at data[len] is therefore a heap-buffer-overflow.
The relevant implementations return raw pointers without adding a terminator: PyBytes_AsString(), PyByteArray_AsString(), and PyUnicode_AsUTF8AndSize().
The bytearray case was raised during review of PR #8009, but dismissed as intentional. However, I'm not sure whether this being a soundness issue was raised at that time.
Reproducer
Save as capi_trailing_null.c:
#include <Python.h>
#include <stdio.h>
int main(void)
{
PyObject *bytes, *bytearray, *unicode;
char *bytes_data, *bytearray_data;
const char *unicode_data;
Py_ssize_t bytes_len, bytearray_len, unicode_len;
unsigned char bytes_trailing, bytearray_trailing, unicode_trailing;
Py_Initialize();
bytes = PyBytes_FromStringAndSize("abc", 3);
bytearray = PyByteArray_FromStringAndSize("abc", 3);
unicode = PyUnicode_FromStringAndSize("abc", 3);
if (!bytes || !bytearray || !unicode)
return 2;
bytes_data = PyBytes_AsString(bytes);
bytearray_data = PyByteArray_AsString(bytearray);
unicode_data = PyUnicode_AsUTF8AndSize(unicode, &unicode_len);
bytes_len = PyBytes_Size(bytes);
bytearray_len = PyByteArray_Size(bytearray);
if (!bytes_data || !bytearray_data || !unicode_data ||
bytes_len < 0 || bytearray_len < 0)
return 2;
bytes_trailing = (unsigned char)bytes_data[bytes_len];
bytearray_trailing = (unsigned char)bytearray_data[bytearray_len];
unicode_trailing = (unsigned char)unicode_data[unicode_len];
printf("bytes: data[len] = 0x%02x\n", bytes_trailing);
printf("bytearray: data[len] = 0x%02x\n", bytearray_trailing);
printf("unicode UTF-8: data[len] = 0x%02x\n", unicode_trailing);
Py_Finalize();
return bytes_trailing || bytearray_trailing || unicode_trailing;
}Expected
With CPython:
$ cc -fsanitize=address -fno-omit-frame-pointer $(python3-config --cflags) capi_trailing_null.c $(python3-config --embed --ldflags) -o trailing-null-cpython
$ ASAN_OPTIONS=detect_leaks=0 ./trailing-null-cpython
bytes: data[len] = 0x00
bytearray: data[len] = 0x00
unicode UTF-8: data[len] = 0x00
Actual
From a RustPython checkout on macOS (use .so instead of .dylib on Linux):
$ cargo build -p rustpython-capi
$ cc -fsanitize=address -fno-omit-frame-pointer $(python3-config --includes) capi_trailing_null.c target/debug/librustpython_capi.dylib -o trailing-null-rustpython
$ ASAN_OPTIONS=detect_leaks=0 ./trailing-null-rustpython
==...==ERROR: AddressSanitizer: heap-buffer-overflow
...
0x... is located 0 bytes after 3-byte region [0x...,0x...)
Python Documentation
CPython documents the trailing byte for PyBytes_AsString(), PyByteArray_AsString(), and PyUnicode_AsUTF8AndSize().
Summary
The C API documents that the buffers returned by
PyBytes_AsString(),PyByteArray_AsString(), andPyUnicode_AsUTF8AndSize()to contain an extra trailing NUL byte. RustPython instead returns pointers to the objects' exact-length Rust buffers. Reading the documented byte atdata[len]is therefore a heap-buffer-overflow.The relevant implementations return raw pointers without adding a terminator:
PyBytes_AsString(),PyByteArray_AsString(), andPyUnicode_AsUTF8AndSize().The bytearray case was raised during review of PR #8009, but dismissed as intentional. However, I'm not sure whether this being a soundness issue was raised at that time.
Reproducer
Save as
capi_trailing_null.c:Expected
With CPython:
Actual
From a RustPython checkout on macOS (use
.soinstead of.dylibon Linux):Python Documentation
CPython documents the trailing byte for
PyBytes_AsString(),PyByteArray_AsString(), andPyUnicode_AsUTF8AndSize().