[CUDA] Upgrade cuDNN to 9.15.1 for CUDA 13 builds by eqy · Pull Request #169412 · pytorch/pytorch

eqy · 2025-12-02T21:28:21Z

Opening this PR for testing...

Note that we are proposing 9.15 instead of 9.16 as we have not had sufficient signal on 9.16 internally

NS: Added hacky workaround to install 9.15.1 for torchbench testing

cc @csarofeen @ptrblck @xwang233 @nWEIdia

pytorch-bot · 2025-12-02T21:28:29Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/169412

📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓ Need help or want to give feedback on the CI? Visit the bot commands wiki

Note: Links to docs will display an error until the docs builds have been completed.

❌ 29 New Failures, 4 Cancelled Jobs, 9 Pending, 1 Unrelated Failure

As of commit 8978cc8 with merge base a01538f ():

NEW FAILURES - The following jobs have failed:

inductor / unit-test / inductor-cpu-core-test (3.13) / test (inductor_core, 1, 2, linux.c7i.2xlarge) (gh)
Process completed with exit code 9.
inductor / unit-test / inductor-cpu-core-test (3.13) / test (inductor_core, 2, 2, linux.c7i.2xlarge) (gh)
Process completed with exit code 9.
inductor-periodic / rocm-periodic-dynamo-benchmarks-build / build (gh)
Limited CI on H100 / linux-jammy-cuda12_8-py3_10-gcc11-sm90-FA3-ABI-stable-test / test (gh)
Process completed with exit code 9.
linux-binary-libtorch / libtorch-rocm7_0-shared-with-deps-release-test (gh)
Unable to download artifact(s): Artifact not found for name: libtorch-rocm7_0-shared-with-deps-release
linux-binary-libtorch / libtorch-rocm7_1-shared-with-deps-release-test (gh)
Unable to download artifact(s): Artifact not found for name: libtorch-rocm7_1-shared-with-deps-release
linux-binary-manywheel / manywheel-py3_10-rocm7_0-test (gh)
Unable to download artifact(s): Artifact not found for name: manywheel-py3_10-rocm7_0
linux-binary-manywheel / manywheel-py3_10-rocm7_1-test (gh)
Unable to download artifact(s): Artifact not found for name: manywheel-py3_10-rocm7_1
linux-binary-manywheel / manywheel-py3_10-xpu-test (gh)
Unable to download artifact(s): Artifact not found for name: manywheel-py3_10-xpu
linux-binary-manywheel / manywheel-py3_11-rocm7_0-test (gh)
Unable to download artifact(s): Artifact not found for name: manywheel-py3_11-rocm7_0
linux-binary-manywheel / manywheel-py3_11-rocm7_1-test (gh)
Unable to download artifact(s): Artifact not found for name: manywheel-py3_11-rocm7_1
linux-binary-manywheel / manywheel-py3_11-xpu-test (gh)
Unable to download artifact(s): Artifact not found for name: manywheel-py3_11-xpu
linux-binary-manywheel / manywheel-py3_12-rocm7_0-test (gh)
Unable to download artifact(s): Artifact not found for name: manywheel-py3_12-rocm7_0
linux-binary-manywheel / manywheel-py3_12-rocm7_1-test (gh)
Unable to download artifact(s): Artifact not found for name: manywheel-py3_12-rocm7_1
linux-binary-manywheel / manywheel-py3_12-xpu-test (gh)
Unable to download artifact(s): Artifact not found for name: manywheel-py3_12-xpu
linux-binary-manywheel / manywheel-py3_13-rocm7_0-test (gh)
Unable to download artifact(s): Artifact not found for name: manywheel-py3_13-rocm7_0
linux-binary-manywheel / manywheel-py3_13-rocm7_1-test (gh)
Unable to download artifact(s): Artifact not found for name: manywheel-py3_13-rocm7_1
linux-binary-manywheel / manywheel-py3_13-xpu-test (gh)
Unable to download artifact(s): Artifact not found for name: manywheel-py3_13-xpu
linux-binary-manywheel / manywheel-py3_13t-rocm7_0-test (gh)
Unable to download artifact(s): Artifact not found for name: manywheel-py3_13t-rocm7_0
linux-binary-manywheel / manywheel-py3_13t-rocm7_1-test (gh)
Unable to download artifact(s): Artifact not found for name: manywheel-py3_13t-rocm7_1
linux-binary-manywheel / manywheel-py3_13t-xpu-test (gh)
Unable to download artifact(s): Artifact not found for name: manywheel-py3_13t-xpu
linux-binary-manywheel / manywheel-py3_14-rocm7_0-test (gh)
Unable to download artifact(s): Artifact not found for name: manywheel-py3_14-rocm7_0
linux-binary-manywheel / manywheel-py3_14-rocm7_1-test (gh)
Unable to download artifact(s): Artifact not found for name: manywheel-py3_14-rocm7_1
linux-binary-manywheel / manywheel-py3_14-xpu-test (gh)
Unable to download artifact(s): Artifact not found for name: manywheel-py3_14-xpu
linux-binary-manywheel / manywheel-py3_14t-rocm7_0-test (gh)
Unable to download artifact(s): Artifact not found for name: manywheel-py3_14t-rocm7_0
linux-binary-manywheel / manywheel-py3_14t-rocm7_1-test (gh)
Unable to download artifact(s): Artifact not found for name: manywheel-py3_14t-rocm7_1
linux-binary-manywheel / manywheel-py3_14t-xpu-test (gh)
Unable to download artifact(s): Artifact not found for name: manywheel-py3_14t-xpu
pull / linux-docs / build-docs-cpp-false (gh)
Process completed with exit code 9.
trunk / cross-compile-linux-test / test (aoti_cross_compile_for_windows, 1, 1, linux.g6.4xlarge.experimental.nvidia.gpu, win-vs2022-... (gh)
Process completed with exit code 9.

CANCELLED JOBS - The following jobs were cancelled. Please retry:

inductor / unit-test / inductor-cpu-core-test (3.11) / test (inductor_core, 1, 2, linux.c7i.2xlarge) (gh)
##[error]The operation was canceled.
inductor / unit-test / inductor-cpu-core-test (3.11) / test (inductor_core, 2, 2, linux.c7i.2xlarge) (gh)
##[error]The operation was canceled.
inductor / unit-test / inductor-cpu-core-test (3.12) / test (inductor_core, 1, 2, linux.c7i.2xlarge) (gh)
##[error]The operation was canceled.
inductor / unit-test / inductor-cpu-core-test (3.12) / test (inductor_core, 2, 2, linux.c7i.2xlarge) (gh)
##[error]The operation was canceled.

UNSTABLE - The following job is marked as unstable, possibly due to flakiness on trunk:

pull / linux-docs / build-docs-python-false (gh) (#111256)
Process completed with exit code 9.

This comment was automatically generated by Dr. CI and updates every 15 minutes.

eqy · 2025-12-05T20:12:20Z

@pytorchmergebot rebase

pytorchmergebot · 2025-12-05T20:13:51Z

@pytorchbot started a rebase job onto refs/remotes/origin/viable/strict. Check the current status here

pytorchmergebot · 2025-12-05T20:13:54Z

Successfully rebased cuda13cudnn915 onto refs/remotes/origin/viable/strict, please pull locally before adding more changes (for example, via git checkout cuda13cudnn915 && git pull --rebase)

tinglvv · 2025-12-08T17:59:15Z

@pytorchbot rebase

pytorchmergebot · 2025-12-08T18:00:52Z

@pytorchbot started a rebase job onto refs/remotes/origin/viable/strict. Check the current status here

pytorchmergebot · 2025-12-08T18:00:55Z

Successfully rebased cuda13cudnn915 onto refs/remotes/origin/viable/strict, please pull locally before adding more changes (for example, via git checkout cuda13cudnn915 && git pull --rebase)

tinglvv · 2025-12-08T18:03:39Z

@pytorchbot rebase -b main

pytorchmergebot · 2025-12-08T18:05:22Z

@pytorchbot started a rebase job onto refs/remotes/origin/main. Check the current status here

pytorchmergebot · 2025-12-08T18:05:25Z

Successfully rebased cuda13cudnn915 onto refs/remotes/origin/main, please pull locally before adding more changes (for example, via git checkout cuda13cudnn915 && git pull --rebase)

eqy · 2025-12-09T00:31:54Z

@pytorchmergebot rebase -b main

pytorchmergebot · 2025-12-09T00:33:26Z

@pytorchbot started a rebase job onto refs/remotes/origin/main. Check the current status here

pytorchmergebot · 2025-12-09T00:33:29Z

Successfully rebased cuda13cudnn915 onto refs/remotes/origin/main, please pull locally before adding more changes (for example, via git checkout cuda13cudnn915 && git pull --rebase)

eqy · 2025-12-09T15:30:00Z

@pytorchmergebot rebase

pytorchmergebot · 2025-12-09T15:31:32Z

@pytorchbot started a rebase job onto refs/remotes/origin/viable/strict. Check the current status here

atalman · 2025-12-23T02:13:20Z

@pytorchmergebot rebase -b main

pytorchmergebot · 2025-12-23T02:14:53Z

@pytorchbot started a rebase job onto refs/remotes/origin/main. Check the current status here

Add conditional installation for CuDNN based on CUDA version.

pytorchmergebot · 2025-12-23T02:14:56Z

Successfully rebased cuda13cudnn915 onto refs/remotes/origin/main, please pull locally before adding more changes (for example, via git checkout cuda13cudnn915 && git pull --rebase)

malfet · 2025-12-23T04:53:24Z

@pytorchbot merge -f "Let's try again, all other failures were fixed by #151700, and revert didn't really helped"

pytorchmergebot · 2025-12-23T04:55:07Z

Merge started

Your change will be merged immediately since you used the force (-f) flag, bypassing any CI checks (ETA: 1-5 minutes). Please use -f as last resort and instead consider -i/--ignore-current to continue the merge ignoring current failures. This will allow currently pending tests to finish and report signal before the merge.

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

atalman · 2025-12-23T14:10:22Z

@pytorchbot cherry-pick --onto release/2.10 -c critical

pytorchbot · 2025-12-23T14:16:27Z

Cherry picking #169412

Command git -C /home/runner/work/pytorch/pytorch cherry-pick -x bfa6f5e0730dead84017e779e02de6cea768ee33 returned non-zero exit code 1

Auto-merging .github/scripts/generate_binary_build_matrix.py
Auto-merging .github/workflows/generated-linux-aarch64-binary-manywheel-nightly.yml
CONFLICT (content): Merge conflict in .github/workflows/generated-linux-aarch64-binary-manywheel-nightly.yml
Auto-merging .github/workflows/generated-linux-binary-manywheel-nightly.yml
CONFLICT (content): Merge conflict in .github/workflows/generated-linux-binary-manywheel-nightly.yml
error: could not apply bfa6f5e0730... [CUDA] Upgrade cuDNN to 9.15.1 for CUDA 13 builds (#169412)
hint: After resolving the conflicts, mark them with
hint: "git add/rm <pathspec>", then run
hint: "git cherry-pick --continue".
hint: You can instead skip this commit with "git cherry-pick --skip".
hint: To abort and get back to the state before "git cherry-pick",
hint: run "git cherry-pick --abort".
hint: Disable this message with "git config set advice.mergeConflict false"

Details for Dev Infra team

Raised by workflow job

Opening this PR for testing... Note that we are proposing 9.15 instead of 9.16 as we have not had sufficient signal on 9.16 internally NS: Added hacky workaround to install 9.15.1 for torchbench testing Pull Request resolved: pytorch#169412 Approved by: https://github.com/atalman, https://github.com/malfet Co-authored-by: Nikita Shulga <2453524+malfet@users.noreply.github.com> Co-authored-by: Ting Lu <tingl@nvidia.com>

[CUDA] Upgrade cuDNN to 9.15.1 for CUDA 13 builds (#169412) Opening this PR for testing... Note that we are proposing 9.15 instead of 9.16 as we have not had sufficient signal on 9.16 internally NS: Added hacky workaround to install 9.15.1 for torchbench testing Pull Request resolved: #169412 Approved by: https://github.com/atalman, https://github.com/malfet Co-authored-by: Nikita Shulga <2453524+malfet@users.noreply.github.com> Co-authored-by: Ting Lu <tingl@nvidia.com>

) Opening this PR for testing... Note that we are proposing 9.15 instead of 9.16 as we have not had sufficient signal on 9.16 internally Pull Request resolved: pytorch#169412 Approved by: https://github.com/atalman, https://github.com/malfet

…orch#169412)" This reverts commit b383f6e. Reverted pytorch#169412 on behalf of https://github.com/malfet due to Broke inductor tests, see https://hud.pytorch.org/hud/pytorch/pytorch/1d2512a56aeae20636aa427fa45693854a5c4e7b/1?per_page=50&name_filter=inductor_torchbench ([comment](pytorch#169412 (comment)))

Opening this PR for testing... Note that we are proposing 9.15 instead of 9.16 as we have not had sufficient signal on 9.16 internally NS: Added hacky workaround to install 9.15.1 for torchbench testing Pull Request resolved: pytorch#169412 Approved by: https://github.com/atalman, https://github.com/malfet Co-authored-by: Nikita Shulga <2453524+malfet@users.noreply.github.com>

…9412)" This reverts commit 83c187f. Reverted pytorch#169412 on behalf of https://github.com/yf225 due to Seems inductor tests still have error https://hud.pytorch.org/pytorch/pytorch/commit/83c187fe2f1f0ea95816778f4b11f0dd4daa5e98#58591620665-box, we might need to revert it again ([comment](pytorch#169412 (comment)))

Opening this PR for testing... Note that we are proposing 9.15 instead of 9.16 as we have not had sufficient signal on 9.16 internally NS: Added hacky workaround to install 9.15.1 for torchbench testing Pull Request resolved: pytorch#169412 Approved by: https://github.com/atalman, https://github.com/malfet Co-authored-by: Nikita Shulga <2453524+malfet@users.noreply.github.com> Co-authored-by: Ting Lu <tingl@nvidia.com>

…9412)" This reverts commit aadd016. Reverted pytorch#169412 on behalf of https://github.com/huydhn due to Sorry for reverting the change but there seems to cause an import error running vLLM tests ([comment](pytorch#169412 (comment)))

Opening this PR for testing... Note that we are proposing 9.15 instead of 9.16 as we have not had sufficient signal on 9.16 internally NS: Added hacky workaround to install 9.15.1 for torchbench testing Pull Request resolved: pytorch#169412 Approved by: https://github.com/atalman, https://github.com/malfet Co-authored-by: Nikita Shulga <2453524+malfet@users.noreply.github.com> Co-authored-by: Ting Lu <tingl@nvidia.com>

eqy requested review from a team and jeffdaily as code owners December 2, 2025 21:28

eqy added module: cudnn Related to torch.backends.cudnn, and CuDNN support open source ciflow/trunk Trigger trunk jobs on your pull request release notes: cudnn ciflow/h100 ciflow/b200 labels Dec 2, 2025

pytorch-bot bot added the topic: not user facing topic category label Dec 2, 2025

eqy changed the title ~~[WIP][CUDA][cuDNN] Upgrade cuDNN to 9.15.1 for CUDA 13 builds~~ [CUDA][cuDNN] Upgrade cuDNN to 9.15.1 for CUDA 13 builds Dec 5, 2025

pytorchmergebot force-pushed the cuda13cudnn915 branch from ef827d3 to bf635d3 Compare December 5, 2025 20:13

eqy mentioned this pull request Dec 5, 2025

[DO NOT MERGE] Testing cuDNN 9.15 with CUDA 13 #169718

Closed

pytorchmergebot force-pushed the cuda13cudnn915 branch from bf635d3 to fdcd13e Compare December 8, 2025 18:00

pytorchmergebot force-pushed the cuda13cudnn915 branch from fdcd13e to c578cf2 Compare December 8, 2025 18:05

pytorchmergebot force-pushed the cuda13cudnn915 branch from c578cf2 to c1f1aeb Compare December 9, 2025 00:33

atalman added the ciflow/vllm label Dec 23, 2025

malfet and others added 4 commits December 23, 2025 02:14

Implement CuDNN installation for CUDA 13

e9e7a52

Add conditional installation for CuDNN based on CUDA version.

Uninstall cudnn after torchbench install

128f688

reapply the CUDNN 9.15 changes

3d9349e

Update install_inductor_benchmark_deps.sh

8978cc8

pytorchmergebot force-pushed the cuda13cudnn915 branch from c8d24b3 to 8978cc8 Compare December 23, 2025 02:14

pytorchmergebot added the merging label Dec 23, 2025

pytorchmergebot closed this in bfa6f5e Dec 23, 2025

pytorchmergebot removed the merging label Dec 23, 2025

This was referenced Dec 23, 2025

[cherry-pick][CUDA] Upgrade cuDNN to 9.15.1 for CUDA 13 builds #171189

Merged

[v.2.10.0] Release Tracker #170119

Closed

Enabling CUDA graphs in compile triggers CUDA Invalid Memory Access #169849

Closed

atalman mentioned this pull request Jan 15, 2026

Release 2.10 validations checklist and cherry-picks #172576

Closed

Conversation

eqy commented Dec 2, 2025 • edited by malfet Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pytorch-bot bot commented Dec 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/169412

❌ 29 New Failures, 4 Cancelled Jobs, 9 Pending, 1 Unrelated Failure

Uh oh!

eqy commented Dec 5, 2025

Uh oh!

pytorchmergebot commented Dec 5, 2025

Uh oh!

pytorchmergebot commented Dec 5, 2025

Uh oh!

tinglvv commented Dec 8, 2025

Uh oh!

pytorchmergebot commented Dec 8, 2025

Uh oh!

pytorchmergebot commented Dec 8, 2025

Uh oh!

tinglvv commented Dec 8, 2025

Uh oh!

pytorchmergebot commented Dec 8, 2025

Uh oh!

pytorchmergebot commented Dec 8, 2025

Uh oh!

eqy commented Dec 9, 2025

Uh oh!

pytorchmergebot commented Dec 9, 2025

Uh oh!

pytorchmergebot commented Dec 9, 2025

Uh oh!

eqy commented Dec 9, 2025

Uh oh!

pytorchmergebot commented Dec 9, 2025

Uh oh!

atalman commented Dec 23, 2025

Uh oh!

pytorchmergebot commented Dec 23, 2025

Uh oh!

pytorchmergebot commented Dec 23, 2025

Uh oh!

malfet commented Dec 23, 2025

Uh oh!

pytorchmergebot commented Dec 23, 2025

Merge started

Uh oh!

atalman commented Dec 23, 2025

Uh oh!

pytorchbot commented Dec 23, 2025

Cherry picking #169412

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

11 participants

eqy commented Dec 2, 2025 •

edited by malfet

Loading

pytorch-bot bot commented Dec 2, 2025 •

edited

Loading