Skip to content

Commit 46860db

Browse files
serhiy-storchakaserhiy-storchaka
committed
gh-108885: Use subtests for doctest examples run by unittest
Run each example as a subtest in unit tests synthesized by doctest.DocFileSuite() and doctest.DocTestSuite().
1 parent 0d32417 commit 46860db

File tree

5 files changed

+280
-218
lines changed

5 files changed

+280
-218
lines changed

‎Doc/library/doctest.rst‎

Lines changed: 40 additions & 21 deletions
Original file line numberDiff line numberDiff line change
@@ -1043,12 +1043,15 @@ from text files and modules with doctests:
10431043
Convert doctest tests from one or more text files to a
10441044
:class:`unittest.TestSuite`.
10451045

1046-
The returned :class:`unittest.TestSuite` is to be run by the unittest framework
1047-
and runs the interactive examples in each file. If an example in any file
1048-
fails, then the synthesized unit test fails, and a :exc:`failureException`
1049-
exception is raised showing the name of the file containing the test and a
1050-
(sometimes approximate) line number. If all the examples in a file are
1051-
skipped, then the synthesized unit test is also marked as skipped.
1046+
The returned :class:`unittest.TestSuite` is to be run by the unittest
1047+
framework and runs the interactive examples in each file.
1048+
Each file is run as a separate unit test, and each example in a file
1049+
is run as a :ref:`subtest <subtests>`.
1050+
If any example in a file fails, then the synthesized unit test fails.
1051+
The traceback for failure or error contains the name of the file
1052+
containing the test and a (sometimes approximate) line number.
1053+
If all the examples in a file are skipped, then the synthesized unit
1054+
test is also marked as skipped.
10521055

10531056
Pass one or more paths (as strings) to text files to be examined.
10541057

@@ -1078,12 +1081,12 @@ from text files and modules with doctests:
10781081

10791082
Optional argument *setUp* specifies a set-up function for the test suite.
10801083
This is called before running the tests in each file. The *setUp* function
1081-
will be passed a :class:`DocTest` object. The setUp function can access the
1084+
will be passed a :class:`DocTest` object. The *setUp* function can access the
10821085
test globals as the *globs* attribute of the test passed.
10831086

10841087
Optional argument *tearDown* specifies a tear-down function for the test
10851088
suite. This is called after running the tests in each file. The *tearDown*
1086-
function will be passed a :class:`DocTest` object. The setUp function can
1089+
function will be passed a :class:`DocTest` object. The *tearDown* function can
10871090
access the test globals as the *globs* attribute of the test passed.
10881091

10891092
Optional argument *globs* is a dictionary containing the initial global
@@ -1105,16 +1108,22 @@ from text files and modules with doctests:
11051108
The global ``__file__`` is added to the globals provided to doctests loaded
11061109
from a text file using :func:`DocFileSuite`.
11071110

1111+
.. versionchanged:: next
1112+
Run each example as a :ref:`subtest <subtests>`.
1113+
11081114

11091115
.. function:: DocTestSuite(module=None, globs=None, extraglobs=None, test_finder=None, setUp=None, tearDown=None, optionflags=0, checker=None)
11101116

11111117
Convert doctest tests for a module to a :class:`unittest.TestSuite`.
11121118

1113-
The returned :class:`unittest.TestSuite` is to be run by the unittest framework
1114-
and runs each doctest in the module. If any of the doctests fail, then the
1115-
synthesized unit test fails, and a :exc:`failureException` exception is raised
1116-
showing the name of the file containing the test and a (sometimes approximate)
1117-
line number. If all the examples in a docstring are skipped, then the
1119+
The returned :class:`unittest.TestSuite` is to be run by the unittest
1120+
framework and runs each doctest in the module.
1121+
Each docstring is run as a separate unit test, and each example in
1122+
a docstring is run as a :ref:`subtest <subtests>`.
1123+
If any of the doctests fail, then the synthesized unit test fails.
1124+
The traceback for failure or error contains the name of the file
1125+
containing the test and a (sometimes approximate) line number.
1126+
If all the examples in a docstring are skipped, then the
11181127
synthesized unit test is also marked as skipped.
11191128

11201129
Optional argument *module* provides the module to be tested. It can be a module
@@ -1132,19 +1141,16 @@ from text files and modules with doctests:
11321141
drop-in replacement) that is used to extract doctests from the module.
11331142

11341143
Optional arguments *setUp*, *tearDown*, and *optionflags* are the same as for
1135-
function :func:`DocFileSuite` above.
1144+
function :func:`DocFileSuite` above, but they are called for each docstring.
11361145

11371146
This function uses the same search technique as :func:`testmod`.
11381147

11391148
.. versionchanged:: 3.5
11401149
:func:`DocTestSuite` returns an empty :class:`unittest.TestSuite` if *module*
11411150
contains no docstrings instead of raising :exc:`ValueError`.
11421151

1143-
.. exception:: failureException
1144-
1145-
When doctests which have been converted to unit tests by :func:`DocFileSuite`
1146-
or :func:`DocTestSuite` fail, this exception is raised showing the name of
1147-
the file containing the test and a (sometimes approximate) line number.
1152+
.. versionchanged:: next
1153+
Run each example as a :ref:`subtest <subtests>`.
11481154

11491155
Under the covers, :func:`DocTestSuite` creates a :class:`unittest.TestSuite` out
11501156
of :class:`!doctest.DocTestCase` instances, and :class:`!DocTestCase` is a
@@ -1508,7 +1514,7 @@ DocTestRunner objects
15081514
with strings that should be displayed. It defaults to ``sys.stdout.write``. If
15091515
capturing the output is not sufficient, then the display output can be also
15101516
customized by subclassing DocTestRunner, and overriding the methods
1511-
:meth:`report_start`, :meth:`report_success`,
1517+
:meth:`report_skip`, :meth:`report_start`, :meth:`report_success`,
15121518
:meth:`report_unexpected_exception`, and :meth:`report_failure`.
15131519

15141520
The optional keyword argument *checker* specifies the :class:`OutputChecker`
@@ -1533,14 +1539,27 @@ DocTestRunner objects
15331539
:class:`DocTestRunner` defines the following methods:
15341540

15351541

1542+
.. method:: report_skip(out, test, example)
1543+
1544+
Report that the given example was skipped. This method is provided to
1545+
allow subclasses of :class:`DocTestRunner` to customize their output; it
1546+
should not be called directly.
1547+
1548+
*example* is the example about to be processed. *test* is the test
1549+
containing *example*. *out* is the output function that was passed to
1550+
:meth:`DocTestRunner.run`.
1551+
1552+
.. versionadded:: next
1553+
1554+
15361555
.. method:: report_start(out, test, example)
15371556

15381557
Report that the test runner is about to process the given example. This method
15391558
is provided to allow subclasses of :class:`DocTestRunner` to customize their
15401559
output; it should not be called directly.
15411560

15421561
*example* is the example about to be processed. *test* is the test
1543-
*containing example*. *out* is the output function that was passed to
1562+
containing *example*. *out* is the output function that was passed to
15441563
:meth:`DocTestRunner.run`.
15451564

15461565

‎Lib/doctest.py‎

Lines changed: 75 additions & 20 deletions
Original file line numberDiff line numberDiff line change
@@ -101,15 +101,14 @@ def _test():
101101
import re
102102
import sys
103103
import traceback
104+
import types
104105
import unittest
105106
from io import StringIO, IncrementalNewlineDecoder
106107
from collections import namedtuple
107108
import _colorize # Used in doctests
108109
from _colorize import ANSIColors, can_colorize
109110

110111

111-
__unittest = True
112-
113112
class TestResults(namedtuple('TestResults', 'failed attempted')):
114113
def __new__(cls, failed, attempted, *, skipped=0):
115114
results = super().__new__(cls, failed, attempted)
@@ -387,7 +386,7 @@ def __init__(self, out):
387386
self.__out = out
388387
self.__debugger_used = False
389388
# do not play signal games in the pdb
390-
pdb.Pdb.__init__(self, stdout=out, nosigint=True)
389+
super().__init__(stdout=out, nosigint=True)
391390
# still use input() to get user input
392391
self.use_rawinput = 1
393392

@@ -1280,6 +1279,11 @@ def __init__(self, checker=None, verbose=None, optionflags=0):
12801279
# Reporting methods
12811280
#/////////////////////////////////////////////////////////////////
12821281

1282+
def report_skip(self, out, test, example):
1283+
"""
1284+
Report that the given example was skipped.
1285+
"""
1286+
12831287
def report_start(self, out, test, example):
12841288
"""
12851289
Report that the test runner is about to process the given
@@ -1377,6 +1381,8 @@ def __run(self, test, compileflags, out):
13771381

13781382
# If 'SKIP' is set, then skip this example.
13791383
if self.optionflags & SKIP:
1384+
if not quiet:
1385+
self.report_skip(out, test, example)
13801386
skips += 1
13811387
continue
13821388

@@ -2275,12 +2281,63 @@ def set_unittest_reportflags(flags):
22752281
return old
22762282

22772283

2284+
class _DocTestCaseRunner(DocTestRunner):
2285+
2286+
def __init__(self, *args, test_case, test_result, **kwargs):
2287+
super().__init__(*args, **kwargs)
2288+
self._test_case = test_case
2289+
self._test_result = test_result
2290+
self._examplenum = 0
2291+
2292+
def _subTest(self):
2293+
subtest = unittest.case._SubTest(self._test_case, str(self._examplenum), {})
2294+
self._examplenum += 1
2295+
return subtest
2296+
2297+
def report_skip(self, out, test, example):
2298+
unittest.case._addSkip(self._test_result, self._subTest(), '')
2299+
2300+
def report_success(self, out, test, example, got):
2301+
self._test_result.addSubTest(self._test_case, self._subTest(), None)
2302+
2303+
def report_unexpected_exception(self, out, test, example, exc_info):
2304+
tb = self._add_traceback(exc_info[2], test, example)
2305+
exc_info = (*exc_info[:2], tb)
2306+
self._test_result.addSubTest(self._test_case, self._subTest(), exc_info)
2307+
2308+
def report_failure(self, out, test, example, got):
2309+
msg = ('Failed example:\n' + _indent(example.source) +
2310+
self._checker.output_difference(example, got, self.optionflags).rstrip('\n'))
2311+
exc = self._test_case.failureException(msg)
2312+
tb = self._add_traceback(None, test, example)
2313+
exc_info = (type(exc), exc, tb)
2314+
self._test_result.addSubTest(self._test_case, self._subTest(), exc_info)
2315+
2316+
def _add_traceback(self, traceback, test, example):
2317+
if test.lineno is None or example.lineno is None:
2318+
lineno = None
2319+
else:
2320+
lineno = test.lineno + example.lineno + 1
2321+
return types.SimpleNamespace(
2322+
tb_frame = types.SimpleNamespace(
2323+
f_globals=test.globs,
2324+
f_code=types.SimpleNamespace(
2325+
co_filename=test.filename,
2326+
co_name=test.name,
2327+
),
2328+
),
2329+
tb_next = traceback,
2330+
tb_lasti = -1,
2331+
tb_lineno = lineno,
2332+
)
2333+
2334+
22782335
class DocTestCase(unittest.TestCase):
22792336

22802337
def __init__(self, test, optionflags=0, setUp=None, tearDown=None,
22812338
checker=None):
22822339

2283-
unittest.TestCase.__init__(self)
2340+
super().__init__()
22842341
self._dt_optionflags = optionflags
22852342
self._dt_checker = checker
22862343
self._dt_test = test
@@ -2304,30 +2361,28 @@ def tearDown(self):
23042361
test.globs.clear()
23052362
test.globs.update(self._dt_globs)
23062363

2364+
def run(self, result=None):
2365+
self._test_result = result
2366+
return super().run(result)
2367+
23072368
def runTest(self):
23082369
test = self._dt_test
2309-
old = sys.stdout
2310-
new = StringIO()
23112370
optionflags = self._dt_optionflags
2371+
result = self._test_result
23122372

23132373
if not (optionflags & REPORTING_FLAGS):
23142374
# The option flags don't include any reporting flags,
23152375
# so add the default reporting flags
23162376
optionflags |= _unittest_reportflags
2377+
if getattr(result, 'failfast', False):
2378+
optionflags |= FAIL_FAST
23172379

2318-
runner = DocTestRunner(optionflags=optionflags,
2319-
checker=self._dt_checker, verbose=False)
2320-
2321-
try:
2322-
runner.DIVIDER = "-"*70
2323-
results = runner.run(test, out=new.write, clear_globs=False)
2324-
if results.skipped == results.attempted:
2325-
raise unittest.SkipTest("all examples were skipped")
2326-
finally:
2327-
sys.stdout = old
2328-
2329-
if results.failed:
2330-
raise self.failureException(self.format_failure(new.getvalue().rstrip('\n')))
2380+
runner = _DocTestCaseRunner(optionflags=optionflags,
2381+
checker=self._dt_checker, verbose=False,
2382+
test_case=self, test_result=result)
2383+
results = runner.run(test, clear_globs=False)
2384+
if results.skipped == results.attempted:
2385+
raise unittest.SkipTest("all examples were skipped")
23312386

23322387
def format_failure(self, err):
23332388
test = self._dt_test
@@ -2442,7 +2497,7 @@ def shortDescription(self):
24422497
class SkipDocTestCase(DocTestCase):
24432498
def __init__(self, module):
24442499
self.module = module
2445-
DocTestCase.__init__(self, None)
2500+
super().__init__(None)
24462501

24472502
def setUp(self):
24482503
self.skipTest("DocTestSuite will not work with -O2 and above")

0 commit comments

Comments
 (0)