Skip to content

Don't fail entirely when can't install requirements for a specific benchmark #350

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
mdboom merged 1 commit into from
Jul 19, 2024
Merged

Don't fail entirely when can't install requirements for a specific benchmark #350

mdboom merged 1 commit into from
Jul 19, 2024

Conversation

@mdboom
Copy link
Contributor

@mdboom mdboom commented Jul 19, 2024

When running multiple benchmarks, pyperformance first creates a venv with the dependencies for each benchmark. More specifically, it actually first tries to install the dependencies into a "global" venv (to reduce the number of total venvs created), and if that fails installs into a new venv specific to the benchmark. If something is broken in one of the dependencies (usually it's because upstream CPython broke backward compatibility on purpose and the dependency no longer compiles), it fails on both attempts, and pyperformance just moves along to the next benchmark.

However, a recent change to setuptools now leaves the package cache in a bad state on the first attempt such that installing wheel on the second attempt fails. This creates an exception where one wasn't expected and the whole pyperformance run comes to a halt.

I think it's safe to expect that creating the venv fails in this way on the second attempt and we should handle it gracefully.

I also plan to investigate the setuptools change further and report there, but in the meantime, I think it makes sense to make pyperformance more robust to that.

==================================================
(58/81) creating venv for benchmark (pytorch_alexnet_inference)

(trying common venv first)
Installing requirements into the virtual environment C:\actions-runner\_work\benchmarking\benchmarking\venv\cpython3.14-2a0759990ce7-compat-af30061f91bc
# 'C:\actions-runner\_work\benchmarking\benchmarking\venv\cpython3.14-2a0759990ce7-compat-af30061f91bc\Scripts\python.exe' -m pip install future==0.18.2 numpy==1.19.0 Pillow==8.0.0 torch==1.5.1 torchvision==0.6.1 'pyperf @ git+https://github.com/psf/pyperf@7b9a23adb3ceeb182f58a03e453158dbe081a7e7'
Collecting pyperf@ git+https://github.com/psf/pyperf@7b9a23adb3ceeb182f58a03e453158dbe081a7e7
  Using cached pyperf-2.7.0-py3-none-any.whl
Collecting future==0.18.2
  Using cached future-0.18.2.tar.gz (829 kB)
  Preparing metadata (setup.py): started
  Preparing metadata (setup.py): finished with status 'done'
Collecting numpy==1.19.0
  Using cached numpy-1.19.0.zip (7.3 MB)
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'error'
  error: subprocess-exited-with-error
  
  Preparing metadata (pyproject.toml) did not run successfully.
  exit code: 1
  
  [102 lines of output]
  Running from numpy source directory.
  <string>:460: UserWarning: Unrecognized setuptools command, proceeding with generating Cython sources and expanding templates
  C:\Windows\ServiceProfiles\NetworkService\AppData\Local\Temp\pip-install-6s5v8bw7\numpy_987489534d5543d08fa6076fdcc0d42d\tools\cythonize.py:73: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
    required_version = LooseVersion('0.29.14')
  C:\Windows\ServiceProfiles\NetworkService\AppData\Local\Temp\pip-install-6s5v8bw7\numpy_987489534d5543d08fa6076fdcc0d42d\tools\cythonize.py:75: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
    if LooseVersion(cython_version) < required_version:
  performance hint: _common.pyx:275:19: Exception check after calling 'random_func' will always require the GIL to be acquired. Declare 'random_func' as 'noexcept' if you control the definition and you're sure you don't want the function to raise exceptions.
  performance hint: _common.pyx:299:19: Exception check after calling 'random_func' will always require the GIL to be acquired. Declare 'random_func' as 'noexcept' if you control the definition and you're sure you don't want the function to raise exceptions.
  performance hint: _common.pyx:322:50: Exception check after calling 'random_func' will always require the GIL to be acquired. Declare 'random_func' as 'noexcept' if you control the definition and you're sure you don't want the function to raise exceptions.
  performance hint: _common.pyx:426:31: Exception check after calling 'f' will always require the GIL to be acquired. Declare 'f' as 'noexcept' if you control the definition and you're sure you don't want the function to raise exceptions.
  performance hint: _common.pyx:465:31: Exception check after calling 'f' will always require the GIL to be acquired. Declare 'f' as 'noexcept' if you control the definition and you're sure you don't want the function to raise exceptions.
  performance hint: _common.pyx:509:31: Exception check after calling 'f' will always require the GIL to be acquired. Declare 'f' as 'noexcept' if you control the definition and you're sure you don't want the function to raise exceptions.
  performance hint: _common.pyx:592:36: Exception check after calling 'f0' will always require the GIL to be acquired. Declare 'f0' as 'noexcept' if you control the definition and you're sure you don't want the function to raise exceptions.
  performance hint: _common.pyx:596:36: Exception check after calling 'f1' will always require the GIL to be acquired. Declare 'f1' as 'noexcept' if you control the definition and you're sure you don't want the function to raise exceptions.
  performance hint: _common.pyx:600:36: Exception check after calling 'f2' will always require the GIL to be acquired. Declare 'f2' as 'noexcept' if you control the definition and you're sure you don't want the function to raise exceptions.
  performance hint: _common.pyx:604:36: Exception check after calling 'f3' will always require the GIL to be acquired. Declare 'f3' as 'noexcept' if you control the definition and you're sure you don't want the function to raise exceptions.
  performance hint: _common.pyx:638:31: Exception check after calling 'f' will always require the GIL to be acquired. Declare 'f' as 'noexcept' if you control the definition and you're sure you don't want the function to raise exceptions.
  performance hint: _common.pyx:675:31: Exception check after calling 'f' will always require the GIL to be acquired. Declare 'f' as 'noexcept' if you control the definition and you're sure you don't want the function to raise exceptions.
  performance hint: _common.pyx:712:63: Exception check after calling 'f' will always require the GIL to be acquired. Declare 'f' as 'noexcept' if you control the definition and you're sure you don't want the function to raise exceptions.
  performance hint: _common.pyx:754:31: Exception check after calling 'f' will always require the GIL to be acquired. Declare 'f' as 'noexcept' if you control the definition and you're sure you don't want the function to raise exceptions.
  performance hint: _common.pyx:785:31: Exception check after calling 'f' will always require the GIL to be acquired. Declare 'f' as 'noexcept' if you control the definition and you're sure you don't want the function to raise exceptions.
  performance hint: _common.pyx:903:40: Exception check after calling 'f0' will always require the GIL to be acquired. Declare 'f0' as 'noexcept' if you control the definition and you're sure you don't want the function to raise exceptions.
  performance hint: _common.pyx:907:40: Exception check after calling 'fd' will always require the GIL to be acquired. Declare 'fd' as 'noexcept' if you control the definition and you're sure you don't want the function to raise exceptions.
  performance hint: _common.pyx:911:41: Exception check after calling 'fdd' will always require the GIL to be acquired. Declare 'fdd' as 'noexcept' if you control the definition and you're sure you don't want the function to raise exceptions.
  performance hint: _common.pyx:916:40: Exception check after calling 'fi' will always require the GIL to be acquired. Declare 'fi' as 'noexcept' if you control the definition and you're sure you don't want the function to raise exceptions.
  performance hint: _common.pyx:920:41: Exception check after calling 'fdi' will always require the GIL to be acquired. Declare 'fdi' as 'noexcept' if you control the definition and you're sure you don't want the function to raise exceptions.
  performance hint: _common.pyx:924:38: Exception check after calling 'fiii' will always require the GIL to be acquired. Declare 'fiii' as 'noexcept' if you control the definition and you're sure you don't want the function to raise exceptions.
  performance hint: _common.pyx:960:31: Exception check after calling 'f' will always require the GIL to be acquired. Declare 'f' as 'noexcept' if you control the definition and you're sure you don't want the function to raise exceptions.
  performance hint: _common.pyx:1002:32: Exception check after calling 'f1' will always require the GIL to be acquired. Declare 'f1' as 'noexcept' if you control the definition and you're sure you don't want the function to raise exceptions.
  performance hint: _generator.pyx:706:41: Exception check after calling '_shuffle_int' will always require the GIL to be acquired.
  Possible solutions:
          1. Declare '_shuffle_int' as 'noexcept' if you control the definition and you're sure you don't want the function to raise exceptions.
          2. Use an 'int' return type on '_shuffle_int' to allow an error code to be returned.
  performance hint: _generator.pyx:735:45: Exception check after calling '_shuffle_int' will always require the GIL to be acquired.
  Possible solutions:
          1. Declare '_shuffle_int' as 'noexcept' if you control the definition and you're sure you don't want the function to raise exceptions.
          2. Use an 'int' return type on '_shuffle_int' to allow an error code to be returned.
  
  Error compiling Cython file:
  ------------------------------------------------------------
  ...
          for i in range(1, RK_STATE_LEN):
              self.rng_state.key[i] = val[i]
          self.rng_state.pos = i
  
          self._bitgen.state = &self.rng_state
          self._bitgen.next_uint64 = &mt19937_uint64
                                     ^
  ------------------------------------------------------------
  
  _mt19937.pyx:138:35: Cannot assign type 'uint64_t (*)(void *) except? -1 nogil' to 'uint64_t (*)(void *) noexcept nogil'. Exception values are incompatible. Suggest adding 'noexcept' to the type of the value being assigned.
  Processing numpy/random\_bounded_integers.pxd.in
  Processing numpy/random\bit_generator.pyx
  Processing numpy/random\mtrand.pyx
  Processing numpy/random\_bounded_integers.pyx.in
  Processing numpy/random\_common.pyx
  Processing numpy/random\_generator.pyx
  Processing numpy/random\_mt19937.pyx
  Traceback (most recent call last):
    File "C:\Windows\ServiceProfiles\NetworkService\AppData\Local\Temp\pip-install-6s5v8bw7\numpy_987489534d5543d08fa6076fdcc0d42d\tools\cythonize.py", line 235, in <module>
      main()
      ~~~~^^
    File "C:\Windows\ServiceProfiles\NetworkService\AppData\Local\Temp\pip-install-6s5v8bw7\numpy_987489534d5543d08fa6076fdcc0d42d\tools\cythonize.py", line 231, in main
      find_process_files(root_dir)
      ~~~~~~~~~~~~~~~~~~^^^^^^^^^^
    File "C:\Windows\ServiceProfiles\NetworkService\AppData\Local\Temp\pip-install-6s5v8bw7\numpy_987489534d5543d08fa6076fdcc0d42d\tools\cythonize.py", line 222, in find_process_files
      process(root_dir, fromfile, tofile, function, hash_db)
      ~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "C:\Windows\ServiceProfiles\NetworkService\AppData\Local\Temp\pip-install-6s5v8bw7\numpy_987489534d5543d08fa6076fdcc0d42d\tools\cythonize.py", line 188, in process
      processor_function(fromfile, tofile)
      ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^
    File "C:\Windows\ServiceProfiles\NetworkService\AppData\Local\Temp\pip-install-6s5v8bw7\numpy_987489534d5543d08fa6076fdcc0d42d\tools\cythonize.py", line 77, in process_pyx
      subprocess.check_call(
      ~~~~~~~~~~~~~~~~~~~~~^
          [sys.executable, '-m', 'cython'] + flags + ["-o", tofile, fromfile])
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "C:\actions-runner\_work\benchmarking\benchmarking\cpython\Lib\subprocess.py", line 419, in check_call
      raise CalledProcessError(retcode, cmd)
  subprocess.CalledProcessError: Command '['C:\\actions-runner\\_work\\benchmarking\\benchmarking\\venv\\cpython3.14-2a0759990ce7-compat-af30061f91bc\\Scripts\\python.exe', '-m', 'cython', '-3', '--fast-fail', '-o', '_mt19937.c', '_mt19937.pyx']' returned non-zero exit status 1.
  Cythonizing sources
  Traceback (most recent call last):
    File "C:\actions-runner\_work\benchmarking\benchmarking\venv\cpython3.14-2a0759990ce7-compat-af30061f91bc\Lib\site-packages\pip\_vendor\pyproject_hooks\_in_process\_in_process.py", line 353, in <module>
      main()
      ~~~~^^
    File "C:\actions-runner\_work\benchmarking\benchmarking\venv\cpython3.14-2a0759990ce7-compat-af30061f91bc\Lib\site-packages\pip\_vendor\pyproject_hooks\_in_process\_in_process.py", line 335, in main
      json_out['return_val'] = hook(**hook_input['kwargs'])
                               ~~~~^^^^^^^^^^^^^^^^^^^^^^^^
    File "C:\actions-runner\_work\benchmarking\benchmarking\venv\cpython3.14-2a0759990ce7-compat-af30061f91bc\Lib\site-packages\pip\_vendor\pyproject_hooks\_in_process\_in_process.py", line 149, in prepare_metadata_for_build_wheel
      return hook(metadata_directory, config_settings)
    File "C:\Windows\ServiceProfiles\NetworkService\AppData\Local\Temp\pip-build-env-cg0x_j4n\overlay\Lib\site-packages\setuptools\build_meta.py", line 368, in prepare_metadata_for_build_wheel
      self.run_setup()
      ~~~~~~~~~~~~~~^^
    File "C:\Windows\ServiceProfiles\NetworkService\AppData\Local\Temp\pip-build-env-cg0x_j4n\overlay\Lib\site-packages\setuptools\build_meta.py", line 497, in run_setup
      super().run_setup(setup_script=setup_script)
      ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "C:\Windows\ServiceProfiles\NetworkService\AppData\Local\Temp\pip-build-env-cg0x_j4n\overlay\Lib\site-packages\setuptools\build_meta.py", line 313, in run_setup
      exec(code, locals())
      ~~~~^^^^^^^^^^^^^^^^
    File "<string>", line 489, in <module>
    File "<string>", line 469, in setup_package
    File "<string>", line 274, in generate_cython
  RuntimeError: Running cythonize failed!
  [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

Encountered error while generating package metadata.

See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.

Notice:  A new release of pip is available: 24.1.1 -> 24.1.2
Notice:  To update, run: C:\actions-runner\_work\benchmarking\benchmarking\venv\cpython3.14-2a0759990ce7-compat-af30061f91bc\Scripts\python.exe -m pip install --upgrade pip
Command failed with exit code 1
(falling back to unique venv)
Creating the virtual environment C:\actions-runner\_work\benchmarking\benchmarking\venv\cpython3.14-2a0759990ce7-compat-af30061f91bc-bm-pytorch_alexnet_inference
# 'C:\actions-runner\_work\benchmarking\benchmarking\cpython\PCBuild\amd64\python.exe' -m venv --without-pip 'C:\actions-runner\_work\benchmarking\benchmarking\venv\cpython3.14-2a0759990ce7-compat-af30061f91bc-bm-pytorch_alexnet_inference'
# 'C:\actions-runner\_work\benchmarking\benchmarking\venv\cpython3.14-2a0759990ce7-compat-af30061f91bc-bm-pytorch_alexnet_inference\Scripts\python.exe' -m ensurepip -v -U
Using pip 24.1.1 from C:\WINDOWS\SERVIC~1\NETWOR~1\AppData\Local\Temp\tmptyeix318\pip-24.1.1-py3-none-any.whl\pip (python 3.14)
Looking in links: c:\WINDOWS\SERVIC~1\NETWOR~1\AppData\Local\Temp\tmptyeix318
Processing c:\windows\servic~1\networ~1\appdata\local\temp\tmptyeix318\pip-24.1.1-py3-none-any.whl
Installing collected packages: pip
Successfully installed pip-24.1.1
# 'C:\actions-runner\_work\benchmarking\benchmarking\venv\cpython3.14-2a0759990ce7-compat-af30061f91bc-bm-pytorch_alexnet_inference\Scripts\python.exe' -m pip install -U 'setuptools>=18.5' wheel
Collecting setuptools>=18.5
  Using cached setuptools-71.0.3-py3-none-any.whl.metadata (6.5 kB)
Collecting wheel
  Using cached wheel-0.43.0-py3-none-any.whl.metadata (2.2 kB)
Using cached setuptools-71.0.3-py3-none-any.whl (2.3 MB)
Using cached wheel-0.43.0-py3-none-any.whl (65 kB)
Installing collected packages: wheel, setuptools
ERROR: Could not install packages due to an OSError: [Errno 2] No such file or directory: 'C:\\actions-runner\\_work\\benchmarking\\benchmarking\\venv\\cpython3.14-2a0759990ce7-compat-af30061f91bc-bm-pytorch_alexnet_inference\\Lib\\site-packages\\pkg_resources\\tests\\data\\my-test-package_unpacked-egg\\my_test_package-1.0-py3.7.egg\\EGG-INFO\\dependency_links.txt'


Notice:  A new release of pip is available: 24.1.1 -> 24.1.2
Notice:  To update, run: C:\actions-runner\_work\benchmarking\benchmarking\venv\cpython3.14-2a0759990ce7-compat-af30061f91bc-bm-pytorch_alexnet_inference\Scripts\python.exe -m pip install --upgrade pip
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "C:\actions-runner\_work\benchmarking\benchmarking\venv\Lib\site-packages\pyperformance\__main__.py", line 2, in <module>
    pyperformance.cli.main()
  File "C:\actions-runner\_work\benchmarking\benchmarking\venv\Lib\site-packages\pyperformance\cli.py", line 328, in main
    _main()
  File "C:\actions-runner\_work\benchmarking\benchmarking\venv\Lib\site-packages\pyperformance\cli.py", line 312, in _main
    cmd_run(options, benchmarks)
  File "C:\actions-runner\_work\benchmarking\benchmarking\venv\Lib\site-packages\pyperformance\commands.py", line 180, in cmd_run
    suite, errors = run_benchmarks(benchmarks, executable, options)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\actions-runner\_work\benchmarking\benchmarking\venv\Lib\site-packages\pyperformance\run.py", line 111, in run_benchmarks
    venv = VenvForBenchmarks.ensure(
           ^^^^^^^^^^^^^^^^^^^^^^^^^
Command failed with exit code 1
  File "C:\actions-runner\_work\benchmarking\benchmarking\venv\Lib\site-packages\pyperformance\venv.py", line 197, in ensure
    return cls.create(
           ^^^^^^^^^^^
  File "C:\actions-runner\_work\benchmarking\benchmarking\venv\Lib\site-packages\pyperformance\venv.py", line 164, in create
    self.ensure_pip(upgrade=upgrade)
  File "C:\actions-runner\_work\benchmarking\benchmarking\venv\Lib\site-packages\pyperformance\_venv.py", line 230, in ensure_pip
    raise RequirementsInstallationFailedError('wheel')
pyperformance._venv.RequirementsInstallationFailedError: wheel
Remove directory C:\actions-runner\_work\benchmarking\benchmarking\venv\cpython3.14-2a0759990ce7-compat-af30061f91bc-bm-pytorch_alexnet_inference

@mdboom mdboom requested a review from ericsnowcurrently July 19, 2024 12:48
ericsnowcurrently
ericsnowcurrently approved these changes Jul 19, 2024
View reviewed changes
Copy link
Member

@ericsnowcurrently ericsnowcurrently left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@mdboom mdboom merged commit a0cc449 into python:main Jul 19, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ericsnowcurrently ericsnowcurrently ericsnowcurrently approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@mdboom @ericsnowcurrently