Skip to content
Merged
132 changes: 74 additions & 58 deletions Doc/library/asyncio-task.rst
Original file line number Diff line number Diff line change
Expand Up @@ -402,69 +402,85 @@ Example::
task2 = tg.create_task(another_coro(...))
print(f"Both tasks have completed now: {task1.result()}, {task2.result()}")

The ``async with`` statement will wait for all tasks in the group to finish.
While waiting, new tasks may still be added to the group
(for example, by passing ``tg`` into one of the coroutines
and calling ``tg.create_task()`` in that coroutine). There is also opportunity to
request termination of the entire task group with ``tg.cancel()``, based on some condition.
Once the last task has finished and the ``async with`` block is exited,
no new tasks may be added to the group.

The first time any of the tasks belonging to the group fails
with an exception other than :exc:`asyncio.CancelledError`,
the remaining tasks in the group are cancelled.
No further tasks can then be added to the group.
At this point, if the body of the ``async with`` statement is still active
(i.e., :meth:`~object.__aexit__` hasn't been called yet),
the task directly containing the ``async with`` statement is also cancelled.
The resulting :exc:`asyncio.CancelledError` will interrupt an ``await``,
but it will not bubble out of the containing ``async with`` statement.

Once all tasks have finished, if any tasks have failed
with an exception other than :exc:`asyncio.CancelledError`,
those exceptions are combined in an
:exc:`ExceptionGroup` or :exc:`BaseExceptionGroup`
(as appropriate; see their documentation)
which is then raised.

Two base exceptions are treated specially:
If any task fails with :exc:`KeyboardInterrupt` or :exc:`SystemExit`,
the task group still cancels the remaining tasks and waits for them,
but then the initial :exc:`KeyboardInterrupt` or :exc:`SystemExit`
is re-raised instead of :exc:`ExceptionGroup` or :exc:`BaseExceptionGroup`.

If the body of the ``async with`` statement exits with an exception
(so :meth:`~object.__aexit__` is called with an exception set),
this is treated the same as if one of the tasks failed:
the remaining tasks are cancelled and then waited for,
and non-cancellation exceptions are grouped into an
exception group and raised.
The exception passed into :meth:`~object.__aexit__`,
unless it is :exc:`asyncio.CancelledError`,
is also included in the exception group.
The same special case is made for
:exc:`KeyboardInterrupt` and :exc:`SystemExit` as in the previous paragraph.
There is an additional special case made only for the body of the
``async with``: if it raises :exc:`GeneratorExit` and none of the
other tasks raise exceptions that would be reported, then the
:exc:`GeneratorExit` is reraised.

Task groups are careful not to mix up the internal cancellation used to
"wake up" their :meth:`~object.__aexit__` with cancellation requests
for the task in which they are running made by other parties.
A few points to keep in mind when using task groups:

* The ``async with`` statement will wait for all tasks in the group
to finish. While waiting, new tasks may still be added to the group
(for example, by passing ``tg`` into one of the coroutines and
calling ``tg.create_task()`` in that coroutine); once the last task
has finished and the ``async with`` block is exited, no new tasks
may be added.

* Termination of the entire task group may be requested with
``tg.cancel()``, based on some condition.

* If the group is shut down (e.g. because another task failed) before
a newly created task has started running, the task is cancelled
without its coroutine executing at all, not even to its first
``await``. To guarantee that the coroutine starts, create the task
eagerly with ``eager_start=True`` or use
:func:`asyncio.eager_task_factory`. For example::

async def job():
print("job started") # never printed
try:
await asyncio.sleep(1)
finally:
print("job cleaned up") # never printed

async def main():
async with asyncio.TaskGroup() as tg:
tg.create_task(job())
raise RuntimeError # shuts down the group before job() runs

With ``tg.create_task(job(), eager_start=True)``, ``job()`` runs up
to the ``await``, is cancelled there, and both messages are printed.

When any of the tasks belonging to the group fails with an exception
other than :exc:`asyncio.CancelledError` (or the body of the
``async with`` statement exits with an exception, which is treated
the same way):

* The first time this happens, the remaining tasks in the group are
cancelled and then waited for, and no further tasks can be added to
the group. If the body of the ``async with`` statement is still
active (i.e., :meth:`~object.__aexit__` hasn't been called yet),
the task directly containing the ``async with`` statement is also
cancelled. The resulting :exc:`asyncio.CancelledError` will
interrupt an ``await``, but it will not bubble out of the containing
``async with`` statement.

* Once all tasks have finished, the non-cancellation exceptions --
including the exception the body exited with, unless it is
:exc:`asyncio.CancelledError` -- are combined in an
:exc:`ExceptionGroup` or :exc:`BaseExceptionGroup`
(as appropriate; see their documentation), which is then raised.

* Some exceptions are treated specially: if any task fails with
:exc:`KeyboardInterrupt` or :exc:`SystemExit`, the task group still
cancels the remaining tasks and waits for them, but then the initial
:exc:`KeyboardInterrupt` or :exc:`SystemExit` is re-raised instead
of :exc:`ExceptionGroup` or :exc:`BaseExceptionGroup`.
Additionally, if the body of the ``async with`` statement raises
:exc:`GeneratorExit` and none of the other tasks raise exceptions
that would be reported, the :exc:`GeneratorExit` is re-raised.

Task groups are careful not to mix up the internal cancellation used
to "wake up" their :meth:`~object.__aexit__` with cancellation
requests for the task in which they are running made by other parties.
In particular, when one task group is syntactically nested in another,
and both experience an exception in one of their child tasks simultaneously,
the inner task group will process its exceptions, and then the outer task group
will receive another cancellation and process its own exceptions.
and both experience an exception in one of their child tasks
simultaneously, the inner task group will process its exceptions, and
then the outer task group will receive another cancellation and
process its own exceptions.

In the case where a task group is cancelled externally and also must
raise an :exc:`ExceptionGroup`, it will call the parent task's
:meth:`~asyncio.Task.cancel` method. This ensures that a
:meth:`~asyncio.Task.cancel` method. This ensures that a
:exc:`asyncio.CancelledError` will be raised at the next
:keyword:`await`, so the cancellation is not lost.

Task groups preserve the cancellation count
reported by :meth:`asyncio.Task.cancelling`.
:keyword:`await`, so the cancellation is not lost. Task groups also
preserve the cancellation count reported by
:meth:`asyncio.Task.cancelling`.

.. versionchanged:: 3.13

Expand Down
5 changes: 5 additions & 0 deletions Doc/library/os.rst
Original file line number Diff line number Diff line change
Expand Up @@ -2960,6 +2960,11 @@ features:

.. audit-event:: os.scandir path os.scandir

Sharing a :func:`scandir` iterator between threads will not corrupt the
iterator, but it is subject to :term:`race conditions <race condition>`:
which entries each thread receives is unspecified, and closing the iterator
while another thread is iterating ends that iteration early.

The :func:`scandir` iterator supports the :term:`context manager` protocol
and has the following method:

Expand Down
13 changes: 8 additions & 5 deletions Doc/library/sys.rst
Original file line number Diff line number Diff line change
Expand Up @@ -233,15 +233,15 @@ always available. Unless explicitly noted otherwise, all variables are read-only

.. function:: _clear_type_cache()

Clear the internal type cache. The type cache is used to speed up attribute
and method lookups. Use the function *only* to drop unnecessary references
during reference leak debugging.

This function should be used for internal and specialized purposes only.
This function is a no-op. It used to clear the internal type cache, which
is now implemented per-type.

.. deprecated:: 3.13
Use the more general :func:`_clear_internal_caches` function instead.

.. versionchanged:: 3.16
This function is now a no-op.


.. function:: _clear_internal_caches()

Expand All @@ -250,6 +250,9 @@ always available. Unless explicitly noted otherwise, all variables are read-only

.. versionadded:: 3.13

.. versionchanged:: 3.16
The type cache is no longer cleared, as it is now implemented per-type.


.. function:: _current_frames()

Expand Down
2 changes: 2 additions & 0 deletions Doc/library/sys_path_init.rst
Original file line number Diff line number Diff line change
Expand Up @@ -105,6 +105,8 @@ Please refer to :mod:`site`'s
mechanism, such as :mod:`venv`, that many virtual environment implementations
follow.

.. _sys-path-init-_pth-files:

_pth files
----------

Expand Down
8 changes: 8 additions & 0 deletions Lib/test/test_asyncio/test_futures.py
Original file line number Diff line number Diff line change
Expand Up @@ -255,6 +255,14 @@ def test_future_cancel_message_setter(self):
f.cancel('my message')
f._cancel_message = 'my new message'
self.assertEqual(f._cancel_message, 'my new message')
f._cancel_message = None
self.assertIsNone(f._cancel_message)
f._cancel_message = 'my new message'
if not isinstance(f, futures._PyFuture):
# The C implementation does not support deletion.
with self.assertRaises(AttributeError):
del f._cancel_message
self.assertEqual(f._cancel_message, 'my new message')

# Also check that the value is used for cancel().
with self.assertRaises(asyncio.CancelledError):
Expand Down
22 changes: 21 additions & 1 deletion Lib/test/test_ctypes/test_delattr.py
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
import unittest
from ctypes import POINTER, Structure, c_char, c_int
from ctypes import CDLL, POINTER, Structure, c_char, c_int
from test.support import import_helper


class X(Structure):
Expand All @@ -26,6 +27,25 @@ def test_struct(self):
with self.assertRaises(TypeError):
del struct.foo

def test_raw(self):
chararray = (c_char * 5)()
with self.assertRaises(AttributeError):
del chararray.raw

def test_func_pointer(self):
# Deleting these attributes restores the default.
dll = CDLL(import_helper.import_module('_ctypes_test').__file__)
func = dll._testfunc_i_bhilfd
func.argtypes = [c_int]
func.restype = c_int
func.errcheck = lambda *args: None
del func.argtypes
self.assertIsNone(func.argtypes)
del func.errcheck
self.assertIsNone(func.errcheck)
del func.restype
self.assertIs(func.restype, c_int)


if __name__ == "__main__":
unittest.main()
31 changes: 31 additions & 0 deletions Lib/test/test_curses.py
Original file line number Diff line number Diff line change
Expand Up @@ -1085,6 +1085,37 @@ def test_output_string_pair_restored(self):
curses.A_BOLD)
self.assertEqual(win.attr_get()[1], pair)

def test_cell_embedded_null_chars(self):
# A NUL cannot share a cell with another character: setcchar() takes a
# NUL-terminated string, so the rest of the cell would be dropped.
for text in ['a\0', 'a\0\u0301', 'a\0b', '\0a']:
with self.subTest(text=text):
self.assertRaises(ValueError, curses.complexchar, text)
if WIDE_BUILD:
self.assertRaises(ValueError, self.stdscr.addch, text)

def test_cell_null_char(self):
# A lone NUL is a character like any other, as addch(0) always was.
stdscr = self.stdscr
cell = curses.complexchar('\0')
self.assertEqual(str(cell), '\0')
self.assertEqual(eval(repr(cell), {'curses': curses}), cell)
stdscr.erase()
stdscr.addch(0, 0, 0)
expected = stdscr.instr(0, 0, 4)
for ch in ['\0', cell]:
with self.subTest(ch=ch):
stdscr.erase()
stdscr.addch(0, 0, ch)
self.assertEqual(stdscr.instr(0, 0, 4), expected)
# A cell holding a NUL reads back as the cell that writes it.
win = curses.newwin(3, 8, 0, 0)
win.insch(0, 0, '\0')
self.assertEqual(win.in_wch(0, 0), cell)
# A string of cells cannot hold a NUL: it would end a batch write.
self.assertRaises(ValueError, curses.complexstr, 'a\0b')
self.assertRaises(ValueError, curses.complexstr, '\0')

def test_add_string_behavior(self):
# addstr() advances the cursor past the written text; addnstr()
# writes at most n characters.
Expand Down
2 changes: 1 addition & 1 deletion Lib/test/test_decimal.py
Original file line number Diff line number Diff line change
Expand Up @@ -4331,7 +4331,7 @@ def test_invalid_context(self):

# Attributes cannot be deleted
for attr in ['prec', 'Emax', 'Emin', 'rounding', 'capitals', 'clamp',
'flags', 'traps']:
'flags', 'traps', '_allcr', '_flags', '_traps']:
self.assertRaises(AttributeError, c.__delattr__, attr)

# Invalid attributes
Expand Down
3 changes: 3 additions & 0 deletions Lib/test/test_defaultdict.py
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,9 @@ def test_basic(self):
self.assertIn(42, d2.keys())
self.assertNotIn(12, d2)
self.assertNotIn(12, d2.keys())
d2.default_factory = list
del d2.default_factory
self.assertEqual(d2.default_factory, None)
d2.default_factory = None
self.assertEqual(d2.default_factory, None)
try:
Expand Down
45 changes: 45 additions & 0 deletions Lib/test/test_exceptions.py
Original file line number Diff line number Diff line change
Expand Up @@ -682,6 +682,44 @@ def test_invalid_setattr(self):
msg = "exception context must be None or derive from BaseException"
self.assertRaisesRegex(TE, msg, setattr, exc, '__context__', 1)

def test_object_attributes(self):
# These attributes are implemented as plain object members:
# they accept any object and are reset to None when deleted.
cases = [
(SyntaxError('msgStr'), 'msg'),
(SyntaxError('msgStr'), 'filename'),
(SyntaxError('msgStr'), 'lineno'),
(SyntaxError('msgStr'), 'offset'),
(SyntaxError('msgStr'), 'end_lineno'),
(SyntaxError('msgStr'), 'end_offset'),
(SyntaxError('msgStr'), 'text'),
(SyntaxError('msgStr'), 'print_file_and_line'),
(SyntaxError('msgStr'), '_metadata'),
(ImportError('msgStr'), 'msg'),
(ImportError('msgStr'), 'name'),
(ImportError('msgStr'), 'path'),
(ImportError('msgStr'), 'name_from'),
(SystemExit(1), 'code'),
(StopIteration(), 'value'),
(NameError('msgStr'), 'name'),
(AttributeError('msgStr'), 'name'),
(AttributeError('msgStr'), 'obj'),
(OSError(2, 'msgStr'), 'errno'),
(OSError(2, 'msgStr'), 'strerror'),
(OSError(2, 'msgStr'), 'filename'),
(OSError(2, 'msgStr'), 'filename2'),
(UnicodeDecodeError('utf-8', b'\xff', 0, 1, 'reasonStr'), 'reason'),
]
if sys.platform == 'win32':
cases.append((OSError(2, 'msgStr'), 'winerror'))
for exc, name in cases:
with self.subTest(exc=type(exc).__name__, name=name):
for value in 'strValue', 42, [1, 2], None:
setattr(exc, name, value)
self.assertEqual(getattr(exc, name), value)
delattr(exc, name)
self.assertIsNone(getattr(exc, name))

def test_invalid_delattr(self):
TE = TypeError
try:
Expand Down Expand Up @@ -739,6 +777,13 @@ def testChainingDescriptors(self):
self.assertTrue(e.__suppress_context__)
e.__suppress_context__ = False
self.assertFalse(e.__suppress_context__)
with self.assertRaisesRegex(TypeError,
'attribute value type must be bool'):
e.__suppress_context__ = 1
with self.assertRaisesRegex(TypeError,
"can't delete numeric/char attribute"):
del e.__suppress_context__
self.assertFalse(e.__suppress_context__)

def testKeywordArgs(self):
# test that builtin exception don't take keyword args,
Expand Down
27 changes: 27 additions & 0 deletions Lib/test/test_frame.py
Original file line number Diff line number Diff line change
Expand Up @@ -222,6 +222,33 @@ def test_f_lineno_del_segfault(self):
with self.assertRaises(AttributeError):
del f.f_lineno

def test_f_trace(self):
f, _, _ = self.make_frames()
def tracer(*args):
pass
for value in tracer, 42, None:
f.f_trace = value
self.assertEqual(f.f_trace, value)
f.f_trace = tracer
del f.f_trace
self.assertIsNone(f.f_trace)

def test_f_trace_lines_and_opcodes(self):
f, _, _ = self.make_frames()
for name in 'f_trace_lines', 'f_trace_opcodes':
with self.subTest(name=name):
for value in False, True:
setattr(f, name, value)
self.assertEqual(getattr(f, name), value)
with self.assertRaisesRegex(TypeError,
'attribute value type must be bool'):
setattr(f, name, 1)
with self.assertRaisesRegex(TypeError,
"can't delete numeric/char attribute"):
del f.f_trace_lines
with self.assertRaisesRegex(AttributeError, 'cannot be deleted'):
del f.f_trace_opcodes

def test_f_generator(self):
# Test f_generator in different contexts.

Expand Down
Loading
Loading