[ATen][CUDA][CUBLAS] cublasLtMatmul increase workspace_size #120925

Aidyn-A · 2024-02-29T18:15:28Z

According to the cuBLAS API Reference the recommended workspace size for Hopper is 32 MiB and for the rest architectures 4 MiB. This PR increases the workspace size accordingly. I am not aware of the recommended workspace size for HIP, that is why I am keeping it unchanged.

cc @csarofeen @ptrblck @xwang233

pytorch-bot · 2024-02-29T18:15:31Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/120925

📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit b39bc70 with merge base 7128504 ():
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

aten/src/ATen/cuda/CUDABlas.cpp

eqy

Thanks for the fix!

Aidyn-A · 2024-03-01T00:46:42Z

@pytorchbot merge

pytorchmergebot · 2024-03-01T00:49:17Z

Merge failed

Reason: Approvers from one of the following sets are needed:

superuser (pytorch/metamates)
Core Reviewers (mruberry, lezcano, Skylion007, ngimel, peterbell10)
Core Maintainers (soumith, gchanan, ezyang, dzhulgakov, malfet)

Details for Dev Infra team

Raised by workflow job

Failing merge rule: Core Maintainers

malfet

LGTM, but do you know if this might push memory utilization for some smaller GPUs beyond the limit?

eqy · 2024-03-01T01:54:16Z

We can add an additional qualifier that sm < 80 stick with the original 1 MiB, what do you think @Aidyn-A

Aidyn-A · 2024-03-01T02:06:47Z

A dilemma of perf vs memory. I guess it makes sense to decrease it for old/small GPU. Lemme commit it 👍

Aidyn-A · 2024-03-01T23:30:41Z

@pytorchbot merge

pytorchmergebot · 2024-03-01T23:32:29Z

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

clee2000 · 2024-03-04T22:11:37Z

@pytorchbot revert -m "broke inductor models and caused accuracy regression on nightly dashboard https://hud.pytorch.org/pytorch/pytorch/commit/0a38a6ac8046e4d3f9cfaba86b7ec6517038646f https://github.com/pytorch/pytorch/actions/runs/8118465367/job/22193590228" -c nosignal

pytorchmergebot · 2024-03-04T22:13:17Z

@pytorchbot successfully started a revert job. Check the current status here.
Questions? Feedback? Please reach out to the PyTorch DevX Team

…120925)" This reverts commit 0a38a6a. Reverted #120925 on behalf of https://github.com/clee2000 due to broke inductor models and caused accuracy regression on nightly dashboard https://hud.pytorch.org/pytorch/pytorch/commit/0a38a6ac8046e4d3f9cfaba86b7ec6517038646f https://github.com/pytorch/pytorch/actions/runs/8118465367/job/22193590228 ([comment](#120925 (comment)))

pytorchmergebot · 2024-03-04T22:13:28Z

@Aidyn-A your PR has been successfully reverted.

mortzur · 2024-03-11T21:10:15Z

Aidyn-A A workflow that sets:
os.environ["CUBLAS_WORKSPACE_CONFIG"] = ":4096:8"
for determinism, fails following this change.
Is it possible to check if CUBLAS_WORKSPACE_CONFIG if provided by the user and enforce it if so?

ptrblck · 2024-03-12T16:12:34Z

@mortzur Could you describe what exactly is failing? This PR changes the cublasLt workspace and does not touch cublas workspace handling.

mortzur · 2024-03-12T19:25:21Z

ptrblck you're right, I can set CUBLASLT_WORKSPACE_SIZE to fix the workspace size. no issue here.

ptrblck · 2024-04-11T20:05:47Z

@mortzur Great, thanks for confirming!

@malfet is this solving the internal test issues and is anything else needed to land the PR?

github-actions · 2024-06-10T20:33:41Z

Looks like this PR hasn't been updated in a while so we're going to go ahead and mark this as Stale.
Feel free to remove the Stale label if you feel this was a mistake.
If you are unable to remove the Stale label please contact a maintainer in order to do so.
If you want the bot to never mark this PR stale again, add the no-stale label.
Stale pull requests will automatically be closed after 30 days of inactivity.

Aidyn-A · 2024-07-03T16:23:57Z

@pytorchbot rebase

pytorchmergebot · 2024-07-03T16:25:14Z

@pytorchbot started a rebase job onto refs/remotes/origin/viable/strict. Check the current status here

pytorchmergebot · 2024-07-03T16:25:17Z

Successfully rebased cublaslt_workspace onto refs/remotes/origin/viable/strict, please pull locally before adding more changes (for example, via git checkout cublaslt_workspace && git pull --rebase)

Aidyn-A · 2024-07-08T18:14:03Z

@pytorchbot rebase

pytorchmergebot · 2024-07-08T18:15:24Z

@pytorchbot started a rebase job onto refs/remotes/origin/viable/strict. Check the current status here

pytorchmergebot · 2024-07-08T18:15:42Z

Successfully rebased cublaslt_workspace onto refs/remotes/origin/viable/strict, please pull locally before adding more changes (for example, via git checkout cublaslt_workspace && git pull --rebase)

…es (#145130) As `cuBLAS` workspaces are already per-stream, there shouldn't be kernel execution overlap with `cuBLASLt` kernels. This PR reuses `cuBLAS` workspaces for `cuBLASLt` for the following benefits: + caching (`cuBLAS` workspaces were already cached, so now we get that for `cuBLASLt`) + "free" workspace size bump for `cuBLASLt` `cuBLASLt` workspace sizes were previously smaller than those for `cuBLAS` by default which potentially hurts performance, and we encountered difficulty in increasing the size due to downstream OOMs , see also #120925 + fixes behavior broken behavior with the memtracker; #139442 attempted to handle peaky allocation behavior that broke memtracker equivalence tests but it didn't seem to fully work, here the cached/reused `cuBLAS` workspace seems to fix it + one environment variable to rule them all: `CUBLAS_WORKSPACE_CONFIG` applies directly to `cuBLASLt` without a confusing `CUBLASLT_WORKSPACE_SIZE` that users would also need to consider Pull Request resolved: #145130 Approved by: https://github.com/ngimel

…es (pytorch#145130) As `cuBLAS` workspaces are already per-stream, there shouldn't be kernel execution overlap with `cuBLASLt` kernels. This PR reuses `cuBLAS` workspaces for `cuBLASLt` for the following benefits: + caching (`cuBLAS` workspaces were already cached, so now we get that for `cuBLASLt`) + "free" workspace size bump for `cuBLASLt` `cuBLASLt` workspace sizes were previously smaller than those for `cuBLAS` by default which potentially hurts performance, and we encountered difficulty in increasing the size due to downstream OOMs , see also pytorch#120925 + fixes behavior broken behavior with the memtracker; pytorch#139442 attempted to handle peaky allocation behavior that broke memtracker equivalence tests but it didn't seem to fully work, here the cached/reused `cuBLAS` workspace seems to fix it + one environment variable to rule them all: `CUBLAS_WORKSPACE_CONFIG` applies directly to `cuBLASLt` without a confusing `CUBLASLT_WORKSPACE_SIZE` that users would also need to consider Pull Request resolved: pytorch#145130 Approved by: https://github.com/ngimel

pytorchbot added the open source label Feb 29, 2024

Aidyn-A requested review from eqy and jeffdaily February 29, 2024 18:17

Aidyn-A added module: cublas Problem related to cublas support topic: not user facing topic category labels Feb 29, 2024

eqy reviewed Feb 29, 2024

View reviewed changes

aten/src/ATen/cuda/CUDABlas.cpp Outdated Show resolved Hide resolved

eqy approved these changes Feb 29, 2024

View reviewed changes

pytorch-bot bot added the ciflow/trunk Trigger trunk jobs on your pull request label Mar 1, 2024

pytorchmergebot added the merging label Mar 1, 2024

pytorchmergebot removed the merging label Mar 1, 2024

eqy requested a review from malfet March 1, 2024 00:54

malfet approved these changes Mar 1, 2024

View reviewed changes

pytorchmergebot added the merging label Mar 1, 2024

pytorchmergebot closed this in 0a38a6a Mar 1, 2024

pytorchmergebot added Merged and removed merging labels Mar 1, 2024

pytorchmergebot added the Reverted label Mar 4, 2024

pytorchmergebot reopened this Mar 4, 2024

github-actions bot added the Stale label Jun 10, 2024

pytorchmergebot force-pushed the cublaslt_workspace branch from 8287fa8 to e55fa56 Compare July 3, 2024 16:25

Aidyn-A and others added 6 commits July 8, 2024 18:15

cublasLtMatmul increase workspace_size

67e1378

compare major version only

83ba118

add an additional qualifier that sm < 80

aff319d

fix typo

fdb898d

check memory too

853f7f0

fix totalGlobalMem

b39bc70

pytorchmergebot force-pushed the cublaslt_workspace branch from e55fa56 to b39bc70 Compare July 8, 2024 18:15

github-actions bot closed this Aug 7, 2024

eqy mentioned this pull request Oct 30, 2024

[ROCm] Increase hipBLASLt default workspace size #139300

Closed

eqy mentioned this pull request Jan 18, 2025

[cuBLAS][cuBLASLt] Unify cuBLASLt workspaces with cuBLAS workspaces #145130

Closed

[ATen][CUDA][CUBLAS] cublasLtMatmul increase workspace_size #120925

[ATen][CUDA][CUBLAS] cublasLtMatmul increase workspace_size #120925

Uh oh!

Conversation

Aidyn-A commented Feb 29, 2024 • edited by pytorch-bot bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pytorch-bot bot commented Feb 29, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/120925

✅ No Failures

Uh oh!

Uh oh!

eqy left a comment

Choose a reason for hiding this comment

Uh oh!

Aidyn-A commented Mar 1, 2024

Uh oh!

pytorchmergebot commented Mar 1, 2024

Merge failed

Uh oh!

malfet left a comment

Choose a reason for hiding this comment

Uh oh!

eqy commented Mar 1, 2024

Uh oh!

Aidyn-A commented Mar 1, 2024

Uh oh!

Aidyn-A commented Mar 1, 2024

Uh oh!

pytorchmergebot commented Mar 1, 2024

Merge started

Uh oh!

clee2000 commented Mar 4, 2024

Uh oh!

pytorchmergebot commented Mar 4, 2024

Uh oh!

pytorchmergebot commented Mar 4, 2024

Uh oh!

mortzur commented Mar 11, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ptrblck commented Mar 12, 2024

Uh oh!

mortzur commented Mar 12, 2024

Uh oh!

ptrblck commented Apr 11, 2024

Uh oh!

github-actions bot commented Jun 10, 2024

Uh oh!

Aidyn-A commented Jul 3, 2024

Uh oh!

pytorchmergebot commented Jul 3, 2024

Uh oh!

pytorchmergebot commented Jul 3, 2024

Uh oh!

Aidyn-A commented Jul 8, 2024

Uh oh!

pytorchmergebot commented Jul 8, 2024

Uh oh!

pytorchmergebot commented Jul 8, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

Aidyn-A commented Feb 29, 2024 •

edited by pytorch-bot bot

Loading

pytorch-bot bot commented Feb 29, 2024 •

edited

Loading

mortzur commented Mar 11, 2024 •

edited

Loading