Skip to content

bpo-36686: Improve the documentation of the std* params in loop.subprocess_exec #13586

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
miss-islington merged 7 commits into from
May 27, 2019
Merged

bpo-36686: Improve the documentation of the std* params in loop.subprocess_exec #13586

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
66deb07
Improve the documentation of the std* params in loop.subprocess_exec
sbstp May 26, 2019
3070414
Trim trailing whitespace
sbstp May 26, 2019
d734631
Remove "yet" from the encoding docs
sbstp May 26, 2019
b9c50af
Raise ValueError if text mode is used, add tests for DEVNULL stdio
sbstp May 27, 2019
93f0ab2
Add blurb for news
sbstp May 27, 2019
27eaf08
Split up checks into seperate errors messages
sbstp May 27, 2019
fc25972
Use strict None checks
sbstp May 27, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
68 changes: 44 additions & 24 deletions Doc/library/asyncio-eventloop.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1217,32 +1217,52 @@ async/await code consider using the high-level

Other parameters:

* *stdin*: either a file-like object representing a pipe to be
connected to the subprocess's standard input stream using
:meth:`~loop.connect_write_pipe`, or the
:const:`subprocess.PIPE` constant (default). By default a new
pipe will be created and connected.

* *stdout*: either a file-like object representing the pipe to be
connected to the subprocess's standard output stream using
:meth:`~loop.connect_read_pipe`, or the
:const:`subprocess.PIPE` constant (default). By default a new pipe
will be created and connected.

* *stderr*: either a file-like object representing the pipe to be
connected to the subprocess's standard error stream using
:meth:`~loop.connect_read_pipe`, or one of
:const:`subprocess.PIPE` (default) or :const:`subprocess.STDOUT`
constants.

By default a new pipe will be created and connected. When
:const:`subprocess.STDOUT` is specified, the subprocess' standard
error stream will be connected to the same pipe as the standard
output stream.
* *stdin* can be any of these:

* a file-like object representing a pipe to be connected to the
subprocess's standard input stream using
:meth:`~loop.connect_write_pipe`
* the :const:`subprocess.PIPE` constant (default) which will create a new
pipe and connect it,
* the value ``None`` which will make the subprocess inherit the file
descriptor from this process
* the :const:`subprocess.DEVNULL` constant which indicates that the
special :data:`os.devnull` file will be used

* *stdout* can be any of these:

* a file-like object representing a pipe to be connected to the
subprocess's standard output stream using
:meth:`~loop.connect_write_pipe`
* the :const:`subprocess.PIPE` constant (default) which will create a new
pipe and connect it,
* the value ``None`` which will make the subprocess inherit the file
descriptor from this process
* the :const:`subprocess.DEVNULL` constant which indicates that the
special :data:`os.devnull` file will be used

* *stderr* can be any of these:

* a file-like object representing a pipe to be connected to the
subprocess's standard error stream using
:meth:`~loop.connect_write_pipe`
* the :const:`subprocess.PIPE` constant (default) which will create a new
pipe and connect it,
* the value ``None`` which will make the subprocess inherit the file
descriptor from this process
* the :const:`subprocess.DEVNULL` constant which indicates that the
special :data:`os.devnull` file will be used
* the :const:`subprocess.STDOUT` constant which will connect the standard
error stream to the process' standard output stream

* All other keyword arguments are passed to :class:`subprocess.Popen`
without interpretation, except for *bufsize*, *universal_newlines*
and *shell*, which should not be specified at all.
without interpretation, except for *bufsize*, *universal_newlines*,
*shell*, *text*, *encoding* and *errors*, which should not be specified
at all.

The ``asyncio`` subprocess API does not support decoding the streams
as text. :func:`bytes.decode` can be used to convert the bytes returned
from the stream to text.

See the constructor of the :class:`subprocess.Popen` class
for documentation on other arguments.
Expand Down
19 changes: 18 additions & 1 deletion Lib/asyncio/base_events.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1561,6 +1561,7 @@ async def subprocess_shell(self, protocol_factory, cmd, *,
stderr=subprocess.PIPE,
universal_newlines=False,
shell=True, bufsize=0,
encoding=None, errors=None, text=None,
**kwargs):
if not isinstance(cmd, (bytes, str)):
raise ValueError("cmd must be a string")
Expand All @@ -1570,6 +1571,13 @@ async def subprocess_shell(self, protocol_factory, cmd, *,
raise ValueError("shell must be True")
if bufsize != 0:
raise ValueError("bufsize must be 0")
if text:
raise ValueError("text must be False")
if encoding is not None:
raise ValueError("encoding must be None")
if errors is not None:
raise ValueError("errors must be None")

protocol = protocol_factory()
debug_log = None
if self._debug:
Expand All @@ -1586,13 +1594,22 @@ async def subprocess_shell(self, protocol_factory, cmd, *,
async def subprocess_exec(self, protocol_factory, program, *args,
stdin=subprocess.PIPE, stdout=subprocess.PIPE,
stderr=subprocess.PIPE, universal_newlines=False,
shell=False, bufsize=0, **kwargs):
shell=False, bufsize=0,
encoding=None, errors=None, text=None,
**kwargs):
if universal_newlines:
raise ValueError("universal_newlines must be False")
if shell:
raise ValueError("shell must be False")
if bufsize != 0:
raise ValueError("bufsize must be 0")
if text:
raise ValueError("text must be False")
if encoding is not None:
raise ValueError("encoding must be None")
if errors is not None:
raise ValueError("errors must be None")

popen_args = (program,) + args
for arg in popen_args:
if not isinstance(arg, (str, bytes)):
Expand Down
90 changes: 90 additions & 0 deletions Lib/test/test_asyncio/test_subprocess.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -335,6 +335,63 @@ async def empty_input():
self.assertEqual(output.rstrip(), b'0')
self.assertEqual(exitcode, 0)

def test_devnull_input(self):

async def empty_input():
code = 'import sys; data = sys.stdin.read(); print(len(data))'
proc = await asyncio.create_subprocess_exec(
sys.executable, '-c', code,
stdin=asyncio.subprocess.DEVNULL,
stdout=asyncio.subprocess.PIPE,
stderr=asyncio.subprocess.PIPE,
close_fds=False,
loop=self.loop)
stdout, stderr = await proc.communicate()
exitcode = await proc.wait()
return (stdout, exitcode)

output, exitcode = self.loop.run_until_complete(empty_input())
self.assertEqual(output.rstrip(), b'0')
self.assertEqual(exitcode, 0)

def test_devnull_output(self):

async def empty_output():
code = 'import sys; data = sys.stdin.read(); print(len(data))'
proc = await asyncio.create_subprocess_exec(
sys.executable, '-c', code,
stdin=asyncio.subprocess.PIPE,
stdout=asyncio.subprocess.DEVNULL,
stderr=asyncio.subprocess.PIPE,
close_fds=False,
loop=self.loop)
stdout, stderr = await proc.communicate(b"abc")
exitcode = await proc.wait()
return (stdout, exitcode)

output, exitcode = self.loop.run_until_complete(empty_output())
self.assertEqual(output, None)
self.assertEqual(exitcode, 0)

def test_devnull_error(self):

async def empty_error():
code = 'import sys; data = sys.stdin.read(); print(len(data))'
proc = await asyncio.create_subprocess_exec(
sys.executable, '-c', code,
stdin=asyncio.subprocess.PIPE,
stdout=asyncio.subprocess.PIPE,
stderr=asyncio.subprocess.DEVNULL,
close_fds=False,
loop=self.loop)
stdout, stderr = await proc.communicate(b"abc")
exitcode = await proc.wait()
return (stderr, exitcode)

output, exitcode = self.loop.run_until_complete(empty_error())
self.assertEqual(output, None)
self.assertEqual(exitcode, 0)

def test_cancel_process_wait(self):
# Issue #23140: cancel Process.wait()

Expand Down Expand Up @@ -531,6 +588,39 @@ def test_process_create_warning(self):
with self.assertWarns(DeprecationWarning):
subprocess.Process(transp, proto, loop=self.loop)

def test_create_subprocess_exec_text_mode_fails(self):
async def execute():
with self.assertRaises(ValueError):
await subprocess.create_subprocess_exec(sys.executable,
text=True)

with self.assertRaises(ValueError):
await subprocess.create_subprocess_exec(sys.executable,
encoding="utf-8")

with self.assertRaises(ValueError):
await subprocess.create_subprocess_exec(sys.executable,
errors="strict")

self.loop.run_until_complete(execute())

def test_create_subprocess_shell_text_mode_fails(self):

async def execute():
with self.assertRaises(ValueError):
await subprocess.create_subprocess_shell(sys.executable,
text=True)

with self.assertRaises(ValueError):
await subprocess.create_subprocess_shell(sys.executable,
encoding="utf-8")

with self.assertRaises(ValueError):
await subprocess.create_subprocess_shell(sys.executable,
errors="strict")

self.loop.run_until_complete(execute())


if sys.platform != 'win32':
# Unix
Expand Down
6 changes: 6 additions & 0 deletions Misc/NEWS.d/next/Documentation/2019-05-27-17-28-58.bpo-36686.Zot4sx.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
Improve documentation of the stdin, stdout, and stderr arguments of of the
``asyncio.subprocess_exec`` function to specficy which values are supported.
Also mention that decoding as text is not supported.

Add a few tests to verify that the various values passed to the std*
arguments actually work.