Make runtests.py skip some slow mypyc tests by default (python#8199)

JukkaL · web-flow · commit 89e259f3cbbc · 2020-01-06T11:24:51.000Z
The skipped mypyc run tests are very slow and I believe that they only
rarely fail for changes that don't touch mypyc.

The skipped tests can be run via runtests.py mypyc-extra.

Also explain running tests in some more detail in the the docs.
diff --git a/README.md b/README.md
@@ -114,7 +114,7 @@ Mypy can be integrated into popular IDEs:
   * Using [Syntastic](https://github.com/vim-syntastic/syntastic): in `~/.vimrc` add
     `let g:syntastic_python_checkers=['mypy']`
   * Using [ALE](https://github.com/dense-analysis/ale): should be enabled by default when `mypy` is installed,
-    or can be explicitly enabled by adding `let b:ale_linters = ['mypy']` in `~/vim/ftplugin/python.vim` 
+    or can be explicitly enabled by adding `let b:ale_linters = ['mypy']` in `~/vim/ftplugin/python.vim`
 * Emacs: using [Flycheck](https://github.com/flycheck/) and [Flycheck-mypy](https://github.com/lbolla/emacs-flycheck-mypy)
 * Sublime Text: [SublimeLinter-contrib-mypy](https://github.com/fredcallaway/SublimeLinter-contrib-mypy)
 * Atom: [linter-mypy](https://atom.io/packages/linter-mypy)
@@ -237,7 +237,8 @@ The basic way to run tests:
     $ python2 -m pip install -U typing
     $ ./runtests.py
 
-For more on the tests, see [Test README.md](test-data/unit/README.md)
+For more on the tests, such as how to write tests and how to control
+which tests to run, see [Test README.md](test-data/unit/README.md).
 
 
 Development status
diff --git a/runtests.py b/runtests.py
@@ -44,6 +44,11 @@
                 MYPYC_COMMAND_LINE,
                 ERROR_STREAM]
 
+
+# These must be enabled by explicitly including 'mypyc-extra' on the command line.
+MYPYC_OPT_IN = [MYPYC_RUN,
+                MYPYC_RUN_MULTI]
+
 # We split the pytest run into three parts to improve test
 # parallelization. Each run should have tests that each take a roughly similar
 # time to run.
@@ -65,16 +70,19 @@
          TYPESHED,
          PEP561,
          DAEMON,
-         MYPYC_RUN,
-         MYPYC_RUN_MULTI,
          MYPYC_EXTERNAL,
          MYPYC_COMMAND_LINE,
          ERROR_STREAM]),
+    # Mypyc tests that aren't run by default, since they are slow and rarely
+    # fail for commits that don't touch mypyc
+    'mypyc-extra': 'pytest -k "%s"' % ' or '.join(MYPYC_OPT_IN),
 }
 
 # Stop run immediately if these commands fail
 FAST_FAIL = ['self', 'lint']
 
+DEFAULT_COMMANDS = [cmd for cmd in cmds if cmd != 'mypyc-extra']
+
 assert all(cmd in cmds for cmd in FAST_FAIL)
 
 
@@ -117,10 +125,12 @@ def main() -> None:
 
     if not set(args).issubset(cmds):
         print("usage:", prog, " ".join('[%s]' % k for k in cmds))
+        print()
+        print('Run the given tests. If given no arguments, run everything except mypyc-extra.')
         exit(1)
 
     if not args:
-        args = list(cmds)
+        args = DEFAULT_COMMANDS[:]
 
     status = 0
 
diff --git a/test-data/unit/README.md b/test-data/unit/README.md
@@ -30,7 +30,7 @@ Add the test in this format anywhere in the file:
 with text "abc..."
 - note a space after `E:` and `flags:`
 - `# E:12` adds column number to the expected error
-- use `\` to escape the `#` character and indicate that the rest of the line is part of 
+- use `\` to escape the `#` character and indicate that the rest of the line is part of
 the error message
 - repeating `# E: ` several times in one line indicates multiple expected errors in one line
 - `W: ...` and `N: ...` works exactly like `E:`, but report a warning and a note respectively
@@ -88,29 +88,32 @@ module:
 
     $ python2 -m pip install -U typing
 
-The unit test suites are driven by the `pytest` framework. To run all tests,
+The unit test suites are driven by the `pytest` framework. To run all mypy tests,
 run `pytest` in the mypy repository:
 
-    $ pytest
-
-Note that some tests will be disabled for older python versions.
+    $ pytest mypy
 
 This will run all tests, including integration and regression tests,
-and will type check mypy and verify that all stubs are valid. This may
-take several minutes to run, so you don't want to use this all the time
-while doing development.
+and will verify that all stubs are valid. This may take several minutes to run,
+so you don't want to use this all the time while doing development.
 
 Test suites for individual components are in the files `mypy/test/test*.py`.
 
+Note that some tests will be disabled for older python versions.
+
+If you work on mypyc, you will want to also run mypyc tests:
+
+    $ pytest mypyc
+
 You can run tests from a specific module directly, a specific suite within a
- module, or a test in a suite (even if it's data-driven):
+module, or a test in a suite (even if it's data-driven):
 
     $ pytest mypy/test/testdiff.py
 
     $ pytest mypy/test/testsemanal.py::SemAnalTypeInfoSuite
-    
+
     $ pytest -n0 mypy/test/testargs.py::ArgSuite::test_coherence
-    
+
     $ pytest -n0 mypy/test/testcheck.py::TypeCheckSuite::testCallingVariableWithFunctionType
 
 To control which tests are run and how, you can use the `-k` switch:
@@ -144,10 +147,19 @@ To run the linter:
 
     $ flake8
 
-You can also run all of the above tests together with:
+You can also run all of the above tests using `runtests.py` (this includes
+type checking mypy and linting):
 
     $ python3 runtests.py
 
+By default, this runs everything except some mypyc tests. You can give it
+arguments to control what gets run, such as `self` to run mypy on itself:
+
+    $ python3 runtests.py self
+
+Run `python3 runtests.py mypyc-extra` to run mypyc tests that are not
+enabled by default. This is typically only needed if you work on mypyc.
+
 Many test suites store test case descriptions in text files
 (`test-data/unit/*.test`). The module `mypy.test.data` parses these
 descriptions.
