Skip to content

gh-162054: Use vsnprintf() on all platforms in PyOS_vsnprintf() - #162053

Open
deronnax wants to merge 2 commits into
python:mainfrom
deronnax:mysnprintf-drop-msvc-vsnprintf
Open

deronnax wants to merge 2 commits into
python:mainfrom
deronnax:mysnprintf-drop-msvc-vsnprintf

Conversation

@deronnax

@deronnax deronnax commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

Since Visual Studio 2015, the MSVC runtime provides a C99-conforming vsnprintf(), so the _vsnprintf() special case in PyOS_vsnprintf() is no longer needed. On truncation, _vsnprintf() returned a negative value, unlike the C99 behavior documented for PyOS_vsnprintf().

Also adds a test for the truncation case in test_ctypes and updates the outdated comment.

No existing caller relies on the negative return value for truncation. Marked as draft until the Windows CI results are in.

Fixes #162054. See also #101703 (comment)

Since Visual Studio 2015, the MSVC runtime provides a C99-conforming
vsnprintf(), so the _vsnprintf() special case is no longer needed.
Add a test for the truncation case and update the outdated comment.
@deronnax deronnax changed the title Use vsnprintf() on all platforms in PyOS_vsnprintf() gh-162054: Use vsnprintf() on all platforms in PyOS_vsnprintf() Nov 15, 2026
@deronnax
deronnax marked this pull request as ready for review November 15, 2026 15:49
@aisk aisk added the skip news label Nov 15, 2026
@aisk

aisk commented Nov 15, 2026

Copy link
Copy Markdown
Member

According to the comment changes in this PR, the behavior of PyOS_vsnprintf() has changed on Windows, since it is a public C API, I think this is more like a bugfix than a refactor, so a NEWS entry is required.

@aisk aisk removed the skip news label Nov 15, 2026
@deronnax

deronnax commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor Author

Ah! This wasn't meant as a bug fix: the intent was dead code unnecessary special implementation elimination, since the MSVC-specific _vsnprintf() branch is unneeded with the UCRT. That being said, I agree it can also be seen as a bugfix, as on Windows the return value on truncation now matches the documented behavior. I've added a NEWS entry.

@picnixz

picnixz commented Nov 15, 2026

Copy link
Copy Markdown
Member

Is it dead code for everyone or are there versions where it is NOT dead code or extension modules that could be affected when running on an older platform? If this is possible, I prefer not to change anything.

@deronnax

deronnax commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor Author

"Dead code" was not the right term, sorry: the branch is really executed on MSVC builds, it is just a special case that is not needed anymore.

Starting from python 3.5, PyOS_vsnprintf() is compiled for Windows using Microsoft Visual C++ 14.0, this Visual Studio uses this new UCRT.

The difference is the return value when the output is truncated: -1 before (windows specific), rv >= size now (standard C99).

I don't think it can break much:

The doc says rv >= size means truncated, and rv < 0 means "failed, the cause depends on the platform". So nobody can officially rely on -1 for truncation.
An extension following the doc handles both cases and keeps working (and is even more right on Windows now).
The only code at risk would be some Windows-only code testing rv == -1 for truncation, or a portable code checking only rv < 0, which already uses silently truncated output on Linux and macOS.

That said, I will not insist much to get this merged. I like this change, it provides a (very) small value in code-readiness/cognitive load, though at the expense of a low risk of breaking some wrongly-written extensions. I know CPython value the second over everything.

@picnixz

picnixz commented Nov 16, 2026

Copy link
Copy Markdown
Member

Let me rephrase it as follows: does the current code rely on fragile details and is it still correct with newer MSVC versions? if so, let's keep it that way since there is a slight behavior change (AFAIU)

@deronnax

Copy link
Copy Markdown
Contributor Author

Current code leaks out an undocumented implementation detail on Windows, but it still correct with newer MSVC.

@picnixz

picnixz commented Nov 16, 2026

Copy link
Copy Markdown
Member

Yeah so we don't need to change it I think. The docs are still correct, namely "something went wrong" so I would lean towards keeping it as is unless Chris thinks it is better to change (I will leave that to him).

@deronnax

Copy link
Copy Markdown
Contributor Author

I think it's a cleanup that Python will have to do somewhere in its time (and seemingly so thinks Eryk Sun), but maybe it should be more prepared – maybe at least discussed on the discourse, so everybody is aware of the change. If it does not make it here, I will bring up the discussion on Python's discourse.

@chris-eibl

Copy link
Copy Markdown
Member

Had we started new, I agree we would definitely use the C99 compliant vsnprintf on Windows as well and have parity with Linux. However, I share @picnixz's concern regarding backwards compatibility, especially since PyOS_vsnprintf is part of the stable ABI.

The doc says rv >= size means truncated, and rv < 0 means "failed, the cause depends on the platform". So nobody can officially rely on -1 for truncation.
An extension following the doc handles both cases and keeps working (and is even more right on Windows now).
The only code at risk would be some Windows-only code testing rv == -1 for truncation, or a portable code checking only rv < 0, which already uses silently truncated output on Linux and macOS.

Current code leaks out an undocumented implementation detail on Windows, but it still correct with newer MSVC.

I agree with all your points, but an abi3 extension only written for Windows, not caring about other platforms and sloppily testing via the implementation detail

int rv = PyOS_vsnprintf(buf, sizeof(buf), fmt, va);

if (rv < 0) {
    /* error */
}
else {
    /* Success */
}

would be broken.

Let me rephrase it as follows: does the current code rely on fragile details

No, MS cares a lot about backwards compatibility ...

and is it still correct with newer MSVC versions?

... and will likely keep _vsnprintf including its documented behavior forever.

@deronnax

Copy link
Copy Markdown
Contributor Author

OK. I have a last (bold) suggestion: what about we don't changePyOS_(v)snprintf, and just remove its usage in CPython for straight (v)snprintf?
Looks like the best of both world.

@vstinner

Copy link
Copy Markdown
Member

I'm in a favor of replacing non-standard MSVC _vsnprintf() with C99 standard vsnprintf(). I don't think that this change goes against PyOS_vsnprintf() documentation. This documentation explains well the different return values. Negative return value is documented as:

When rv < 0, the output conversion failed and str[size-1] is '\0' in this case too, but the rest of str is undefined. The exact cause of the error depends on the underlying platform.

Using C99 standard vsnprintf() on Windows, the function no longer returns a negative value if the output is truncated which is a good thing, since truncated strings are also well documented:

When rv >= size, the output conversion was truncated and a buffer with rv + 1 bytes would have been needed to succeed. str[size-1] is '\0' in this case.

@chris-eibl:

PyOS_vsnprintf is part of the stable ABI

It doesn't prevent us from changing PyOS_vsnprintf() implementation, especially if it's to fix the implementation :-)

Hopefully, this function is rarely used. I searched for PyOS_vsnprintf in PyPI top 15,000 projects (at 2026-09-17). I only found 2 projects calling this function:

  • libpff-python (20231205)
  • scipy (1.18.1)

libpff-python raises ValueError if PyOS_vsnprintf() return value is negative, and it doesn't check if the PyOS_vsnprintf() was truncated (it doesn't check rv >= size).

scipy() doesn't check the return value! Code: PyOS_vsnprintf(info, 1024, fmt, ap);

Currently, libpff-python only raises ValueError on Windows if the string is truncated. I suppose that this library was not tested on Windows, or at least truncated strings were not tested on Windows. If MSVC _vsnprintf() if replaced with C99 vsnprintf(), suddenly libpff-python will behave the same on Windows than on other platforms and it will just work as expected.

@chris-eibl:

I agree with all your points, but an abi3 extension only written for Windows, not caring about other platforms and sloppily testing via the implementation detail (...) would be broken.

Correct. But such code doesn't respect PyOS_vsnprintf() documentation. I suggest to only change Python 3.16 (don't backport it) and document the change. If there is impacted code, it can be updated to handle also documented rv >= size code, and so it will work on new and old Python versions.

@picnixz:

does the current code rely on fragile details and is it still correct with newer MSVC versions?

In the 1,093,987 lines of C code of CPython, PyOS_vsnprintf() is only called by a single function: sys_write(), the implementation of PySys_WriteStdout() and PySys_WriteStderr(). It checks properly for failure: if (written < 0 || (size_t)written >= sizeof(buffer)) {.

I'm not sure if I replied correctly to your question.

@@ -0,0 +1,4 @@
:c:func:`PyOS_vsnprintf` now calls :c:func:`!vsnprintf` on Windows, instead of

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
:c:func:`PyOS_vsnprintf` now calls :c:func:`!vsnprintf` on Windows, instead of
:c:func:`PyOS_vsnprintf` now calls standard :c:func:`!vsnprintf` on Windows, instead of

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe even use the standard C99?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

https://peps.python.org/pep-0007/ says that Python requires C11. MSVC doesn't support C99. So if we mention a standard, it would be better to refer to C11.

C11 is basically C99 without features not supported by MSVC.

@@ -0,0 +1,4 @@
:c:func:`PyOS_vsnprintf` now calls :c:func:`!vsnprintf` on Windows, instead of
the non-standard :c:func:`!_vsnprintf`. When the output is truncated, it now
returns the number of characters that would have been written, as on other

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
returns the number of characters that would have been written, as on other
returns the number of bytes that would have been written (excluding the null byte), as on other

Comment on lines +71 to +75
# truncated output: C99 return value and trailing NUL
small = create_string_buffer(b"x" * 8, 8)
rv = PyOS_snprintf(small, sizeof(small), b"Hello from %s", b"ctypes")
self.assertEqual(rv, len(b"Hello from ctypes"))
self.assertEqual(small.raw, b"Hello f\0")

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would prefer to check truncated string on the %s argument rather than on the format string. Also, you should test the invalid format string (-1 return value):

Suggested change
# truncated output: C99 return value and trailing NUL
small = create_string_buffer(b"x" * 8, 8)
rv = PyOS_snprintf(small, sizeof(small), b"Hello from %s", b"ctypes")
self.assertEqual(rv, len(b"Hello from ctypes"))
self.assertEqual(small.raw, b"Hello f\0")
# Test truncated output
small = create_string_buffer(14)
rv = PyOS_snprintf(small, sizeof(small), b"Hello from %s", b"ctypes")
self.assertEqual(rv, len(b"Hello from ctypes"))
self.assertEqual(small.raw, b"Hello from ct\0")
# Test invalid format string
buf = create_string_buffer(14)
rv = PyOS_snprintf(buf, sizeof(buf), b"Hello from %Z", 123)
self.assertEqual(rv, -1)

Usually, we test the Python C API in test_capi. But well, there are already some PyOS_snprintf() tests in test_ctypes, so I'm ok to add a few more there.

@chris-eibl

Copy link
Copy Markdown
Member

I suggest to only change Python 3.16 (don't backport it) and document the change.

Thanks @vstinner for chiming in. I now also think this is the best way to go.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Use vsnprintf() on all platforms in PyOS_vsnprintf()

5 participants