Skip to content

Commit 4beb9fc

Browse files
tgamblintgamblin
committed
tests: improved spack test command line options
Previously, `spack test` automatically passed all of its arguments to `pytest -k` if no options were provided, and to `pytest` if they were. `spack test -l` also provided a list of test filenames, but they didn't really let you completely narrow down which tests you wanted to run. Instead of trying to do our own weird thing, this passes `spack test` args directly to `pytest`, and omits the implicit `-k`. This means we can now run, e.g.: ```console $ spack test spec_syntax.py::TestSpecSyntax::test_ambiguous ``` This wasn't possible before, because we'd pass the fully qualified name to `pytest -k` and get an error. Because `pytest` doesn't have the greatest ability to list tests, I've tweaked the `-l`/`--list`, `-L`/`--list-long`, and `-N`/`--list-names` options to `spack test` so that they help you understand the names better. you can combine these options with `-k` or other arguments to do pretty powerful searches. This one makes it easy to get a list of names so you can run tests in different orders (something I find useful for debugging `pytest` issues): ```console $ spack test --list-names -k "spec and concretize" cmd/env.py::test_concretize_user_specs_together concretize.py::TestConcretize::test_conflicts_in_spec concretize.py::TestConcretize::test_find_spec_children concretize.py::TestConcretize::test_find_spec_none concretize.py::TestConcretize::test_find_spec_parents concretize.py::TestConcretize::test_find_spec_self concretize.py::TestConcretize::test_find_spec_sibling concretize.py::TestConcretize::test_no_matching_compiler_specs concretize.py::TestConcretize::test_simultaneous_concretization_of_specs spec_dag.py::TestSpecDag::test_concretize_deptypes spec_dag.py::TestSpecDag::test_copy_concretized ``` You can combine any list option with keywords: ```console $ spack test --list -k microarchitecture llnl/util/cpu.py modules/lmod.py ``` ```console $ spack test --list-long -k microarchitecture llnl/util/cpu.py:: test_generic_microarchitecture modules/lmod.py::TestLmod:: test_only_generic_microarchitectures_in_root ``` Or just list specific files: ```console $ spack test --list-long cmd/test.py cmd/test.py:: test_list test_list_names_with_pytest_arg test_list_long test_list_with_keywords test_list_long_with_pytest_arg test_list_with_pytest_arg test_list_names ``` Hopefully this stuff will help with debugging test issues. - [x] make `spack test` send args directly to `pytest` instead of trying to do fancy things. - [x] rework `--list`, `--list-long`, and add `--list-names` to make searching for tests easier. - [x] make it possible to mix Spack's list args with `pytest` args (they're just fancy parsing around `pytest --collect-only`) - [x] add docs - [x] add tests - [x] update spack completion
1 parent de73121 commit 4beb9fc

File tree

5 files changed

+266
-62
lines changed

5 files changed

+266
-62
lines changed

‎lib/spack/docs/contribution_guide.rst‎

Lines changed: 64 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -64,6 +64,8 @@ If you take a look in ``$SPACK_ROOT/.travis.yml``, you'll notice that we test
6464
against Python 2.6, 2.7, and 3.4-3.7 on both macOS and Linux. We currently
6565
perform 3 types of tests:
6666

67+
.. _cmd-spack-test:
68+
6769
^^^^^^^^^^
6870
Unit Tests
6971
^^^^^^^^^^
@@ -86,40 +88,83 @@ To run *all* of the unit tests, use:
8688
8789
$ spack test
8890
89-
These tests may take several minutes to complete. If you know you are only
90-
modifying a single Spack feature, you can run a single unit test at a time:
91+
These tests may take several minutes to complete. If you know you are
92+
only modifying a single Spack feature, you can run subsets of tests at a
93+
time. For example, this would run all the tests in
94+
``lib/spack/spack/test/architecture.py``:
9195

9296
.. code-block:: console
9397
94-
$ spack test architecture
98+
$ spack test architecture.py
99+
100+
And this would run the ``test_platform`` test from that file:
101+
102+
.. code-block:: console
95103
96-
This allows you to develop iteratively: make a change, test that change, make
97-
another change, test that change, etc. To get a list of all available unit
98-
tests, run:
104+
$ spack test architecture.py::test_platform
105+
106+
This allows you to develop iteratively: make a change, test that change,
107+
make another change, test that change, etc. We use `pytest
108+
<http://pytest.org/>`_ as our tests fromework, and these types of
109+
arguments are just passed to the ``pytest`` command underneath. See `the
110+
pytest docs
111+
<http://doc.pytest.org/en/latest/usage.html#specifying-tests-selecting-tests>`_
112+
for more details on test selection syntax.
113+
114+
``spack test`` has a few special options that can help you understand
115+
what tests are available. To get a list of all available unit test
116+
files, run:
99117

100118
.. command-output:: spack test --list
119+
:ellipsis: 5
120+
121+
To see a more detailed list of available unit tests, use ``spack test
122+
--list-long``:
123+
124+
.. command-output:: spack test --list-long
125+
:ellipsis: 10
126+
127+
And to see the fully qualified names of all tests, use ``--list-names``:
128+
129+
.. command-output:: spack test --list-names
130+
:ellipsis: 5
131+
132+
You can combine these with ``pytest`` arguments to restrict which tests
133+
you want to know about. For example, to see just the tests in
134+
``architecture.py``:
135+
136+
.. command-output:: spack test --list-long architecture.py
137+
138+
You can also combine any of these options with a ``pytest`` keyword
139+
search. For example, to see the names of all tests that have "spec"
140+
or "concretize" somewhere in their names:
101141

102-
A more detailed list of available unit tests can be found by running
103-
``spack test --long-list``.
142+
.. command-output:: spack test --list-names -k "spec and concretize"
104143

105-
By default, ``pytest`` captures the output of all unit tests. If you add print
106-
statements to a unit test and want to see the output, simply run:
144+
By default, ``pytest`` captures the output of all unit tests, and it will
145+
print any captured output for failed tests. Sometimes it's helpful to see
146+
your output interactively, while the tests run (e.g., if you add print
147+
statements to a unit tests). To see the output *live*, use the ``-s``
148+
argument to ``pytest``:
107149

108150
.. code-block:: console
109151
110-
$ spack test -s -k architecture
152+
$ spack test -s architecture.py::test_platform
111153
112-
Unit tests are crucial to making sure bugs aren't introduced into Spack. If you
113-
are modifying core Spack libraries or adding new functionality, please consider
114-
adding new unit tests or strengthening existing tests.
154+
Unit tests are crucial to making sure bugs aren't introduced into
155+
Spack. If you are modifying core Spack libraries or adding new
156+
functionality, please add new unit tests for your feature, and consider
157+
strengthening existing tests. You will likely be asked to do this if you
158+
submit a pull request to the Spack project on GitHub. Check out the
159+
`pytest docs <http://pytest.org/>`_ and feel free to ask for guidance on
160+
how to write tests!
115161

116162
.. note::
117163

118-
There is also a ``run-unit-tests`` script in ``share/spack/qa`` that
119-
runs the unit tests. Afterwards, it reports back to Codecov with the
120-
percentage of Spack that is covered by unit tests. This script is
121-
designed for Travis CI. If you want to run the unit tests yourself, we
122-
suggest you use ``spack test``.
164+
You may notice the ``share/spack/qa/run-unit-tests`` script in the
165+
repository. This script is designed for Travis CI. It runs the unit
166+
tests and reports coverage statistics back to Codecov. If you want to
167+
run the unit tests yourself, we suggest you use ``spack test``.
123168

124169
^^^^^^^^^^^^
125170
Flake8 Tests

‎lib/spack/docs/developer_guide.rst‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -363,12 +363,12 @@ Developer commands
363363
``spack doc``
364364
^^^^^^^^^^^^^
365365

366-
.. _cmd-spack-test:
367-
368366
^^^^^^^^^^^^^^
369367
``spack test``
370368
^^^^^^^^^^^^^^
371369

370+
See the :ref:`contributor guide section <cmd-spack-test>` on ``spack test``.
371+
372372
.. _cmd-spack-python:
373373

374374
^^^^^^^^^^^^^^^^

‎lib/spack/spack/cmd/test.py‎

Lines changed: 104 additions & 40 deletions
Original file line numberDiff line numberDiff line change
@@ -4,82 +4,153 @@
44
# SPDX-License-Identifier: (Apache-2.0 OR MIT)
55

66
from __future__ import print_function
7+
from __future__ import division
78

9+
import collections
810
import sys
9-
import os
1011
import re
1112
import argparse
1213
import pytest
1314
from six import StringIO
1415

16+
import llnl.util.tty.color as color
1517
from llnl.util.filesystem import working_dir
1618
from llnl.util.tty.colify import colify
1719

1820
import spack.paths
1921

20-
description = "run spack's unit tests"
22+
description = "run spack's unit tests (wrapper around pytest)"
2123
section = "developer"
2224
level = "long"
2325

2426

2527
def setup_parser(subparser):
2628
subparser.add_argument(
2729
'-H', '--pytest-help', action='store_true', default=False,
28-
help="print full pytest help message, showing advanced options")
29-
30-
list_group = subparser.add_mutually_exclusive_group()
31-
list_group.add_argument(
32-
'-l', '--list', action='store_true', default=False,
33-
help="list basic test names")
34-
list_group.add_argument(
35-
'-L', '--long-list', action='store_true', default=False,
36-
help="list the entire hierarchy of tests")
30+
help="show full pytest help, with advanced options")
31+
32+
# extra spack arguments to list tests
33+
list_group = subparser.add_argument_group("listing tests")
34+
list_mutex = list_group.add_mutually_exclusive_group()
35+
list_mutex.add_argument(
36+
'-l', '--list', action='store_const', default=None,
37+
dest='list', const='list', help="list test filenames")
38+
list_mutex.add_argument(
39+
'-L', '--list-long', action='store_const', default=None,
40+
dest='list', const='long', help="list all test functions")
41+
list_mutex.add_argument(
42+
'-N', '--list-names', action='store_const', default=None,
43+
dest='list', const='names', help="list full names of all tests")
44+
45+
# use tests for extension
3746
subparser.add_argument(
3847
'--extension', default=None,
39-
help="run test for a given Spack extension"
40-
)
48+
help="run test for a given spack extension")
49+
50+
# spell out some common pytest arguments, so they'll show up in help
51+
pytest_group = subparser.add_argument_group(
52+
"common pytest arguments (spack test --pytest-help for more details)")
53+
pytest_group.add_argument(
54+
"-s", action='append_const', dest='parsed_args', const='-s',
55+
help="print output while tests run (disable capture)")
56+
pytest_group.add_argument(
57+
"-k", action='store', metavar="EXPRESSION", dest='expression',
58+
help="filter tests by keyword (can also use w/list options)")
59+
pytest_group.add_argument(
60+
"--showlocals", action='append_const', dest='parsed_args',
61+
const='--showlocals', help="show local variable values in tracebacks")
62+
63+
# remainder is just passed to pytest
4164
subparser.add_argument(
42-
'tests', nargs=argparse.REMAINDER,
43-
help="list of tests to run (will be passed to pytest -k)")
65+
'pytest_args', nargs=argparse.REMAINDER, help="arguments for pytest")
4466

4567

46-
def do_list(args, unknown_args):
68+
def do_list(args, extra_args):
4769
"""Print a lists of tests than what pytest offers."""
4870
# Run test collection and get the tree out.
4971
old_output = sys.stdout
5072
try:
5173
sys.stdout = output = StringIO()
52-
pytest.main(['--collect-only'])
74+
pytest.main(['--collect-only'] + extra_args)
5375
finally:
5476
sys.stdout = old_output
5577

56-
# put the output in a more readable tree format.
5778
lines = output.getvalue().split('\n')
58-
output_lines = []
79+
tests = collections.defaultdict(lambda: set())
80+
prefix = []
81+
82+
# collect tests into sections
5983
for line in lines:
6084
match = re.match(r"(\s*)<([^ ]*) '([^']*)'", line)
6185
if not match:
6286
continue
6387
indent, nodetype, name = match.groups()
6488

65-
# only print top-level for short list
66-
if args.list:
67-
if not indent:
68-
output_lines.append(
69-
os.path.basename(name).replace('.py', ''))
70-
else:
71-
print(indent + name)
89+
# strip parametrized tests
90+
if "[" in name:
91+
name = name[:name.index("[")]
92+
93+
depth = len(indent) // 2
7294

73-
if args.list:
74-
colify(output_lines)
95+
if nodetype.endswith("Function"):
96+
key = tuple(prefix)
97+
tests[key].add(name)
98+
else:
99+
prefix = prefix[:depth]
100+
prefix.append(name)
101+
102+
def colorize(c, prefix):
103+
if isinstance(prefix, tuple):
104+
return "::".join(
105+
color.colorize("@%s{%s}" % (c, p))
106+
for p in prefix if p != "()"
107+
)
108+
return color.colorize("@%s{%s}" % (c, prefix))
109+
110+
if args.list == "list":
111+
files = set(prefix[0] for prefix in tests)
112+
color_files = [colorize("B", file) for file in sorted(files)]
113+
colify(color_files)
114+
115+
elif args.list == "long":
116+
for prefix, functions in sorted(tests.items()):
117+
path = colorize("*B", prefix) + "::"
118+
functions = [colorize("c", f) for f in sorted(functions)]
119+
color.cprint(path)
120+
colify(functions, indent=4)
121+
print()
122+
123+
else: # args.list == "names"
124+
all_functions = [
125+
colorize("*B", prefix) + "::" + colorize("c", f)
126+
for prefix, functions in sorted(tests.items())
127+
for f in sorted(functions)
128+
]
129+
colify(all_functions)
130+
131+
132+
def add_back_pytest_args(args, unknown_args):
133+
"""Add parsed pytest args, unknown args, and remainder together.
134+
135+
We add some basic pytest arguments to the Spack parser to ensure that
136+
they show up in the short help, so we have to reassemble things here.
137+
"""
138+
result = args.parsed_args or []
139+
result += unknown_args or []
140+
result += args.pytest_args or []
141+
if args.expression:
142+
result += ["-k", args.expression]
143+
return result
75144

76145

77146
def test(parser, args, unknown_args):
78147
if args.pytest_help:
79148
# make the pytest.main help output more accurate
80149
sys.argv[0] = 'spack test'
81-
pytest.main(['-h'])
82-
return
150+
return pytest.main(['-h'])
151+
152+
# add back any parsed pytest args we need to pass to pytest
153+
pytest_args = add_back_pytest_args(args, unknown_args)
83154

84155
# The default is to test the core of Spack. If the option `--extension`
85156
# has been used, then test that extension.
@@ -91,15 +162,8 @@ def test(parser, args, unknown_args):
91162

92163
# pytest.ini lives in the root of the spack repository.
93164
with working_dir(pytest_root):
94-
# --list and --long-list print the test output better.
95-
if args.list or args.long_list:
96-
do_list(args, unknown_args)
165+
if args.list:
166+
do_list(args, pytest_args)
97167
return
98168

99-
# Allow keyword search without -k if no options are specified
100-
if (args.tests and not unknown_args and
101-
not any(arg.startswith('-') for arg in args.tests)):
102-
return pytest.main(['-k'] + args.tests)
103-
104-
# Just run the pytest command
105-
return pytest.main(unknown_args + args.tests)
169+
return pytest.main(pytest_args)

0 commit comments

Comments
 (0)