consider collective inputs only deallocated on wait() #164839

eellison · 2025-10-07T15:46:10Z

Stack from ghstack (oldest at bottom):

Edit: this is true for reuse on main stream, but not true for reuse on side stream.. going to defer landing since we already limit in flight memory.

cc @H-Huang @awgu @wanchaol @fegin @fduwjj @wz337 @wconstab @d4l3k @pragupta @msaroufim @dcci @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @wenzhe-nrv @jiayisunx @ipiszy @chenyang78 @kadeng @muchulee8 @amjames @chauhang @aakhundov @coconutruben

[ghstack-poisoned]

pytorch-bot · 2025-10-07T15:46:14Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/164839

📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓ Need help or want to give feedback on the CI? Visit the bot commands wiki

Note: Links to docs will display an error until the docs builds have been completed.

❗ 1 Active SEVs

There are 1 currently active SEVs. If your PR is affected, please view them below:

Driver update on H100 and A100 instances

❌ 21 New Failures, 34 Cancelled Jobs

As of commit 3fd1f73 with merge base 7617b11 ():

NEW FAILURES - The following jobs have failed:

inductor / inductor-cpu-test / test (cpu_inductor_torchbench, 2, 2, linux.8xlarge.amx) (gh)
vision_maskrcnn
inductor / inductor-cpu-test / test (dynamic_cpu_inductor_huggingface, 1, 1, linux.8xlarge.amx) (gh)
Process completed with exit code 254.
inductor / inductor-cpu-test / test (dynamic_cpu_inductor_timm, 1, 2, linux.8xlarge.amx) (gh)
Process completed with exit code 254.
inductor / inductor-cpu-test / test (inductor_torchbench_cpu_smoketest_perf, 1, 1, linux.24xl.spr-metal) (gh)
Process completed with exit code 254.
inductor / inductor-test / test (inductor_huggingface, 1, 1, linux.g5.4xlarge.nvidia.gpu) (gh)
Process completed with exit code 254.
inductor / inductor-test / test (inductor_timm, 1, 2, linux.g5.4xlarge.nvidia.gpu) (gh)
Process completed with exit code 254.
inductor / inductor-test / test (inductor_timm, 2, 2, linux.g5.4xlarge.nvidia.gpu) (gh)
Process completed with exit code 254.
inductor / inductor-test / test (inductor_torchbench, 1, 2, linux.g5.4xlarge.nvidia.gpu) (gh)
Process completed with exit code 254.
inductor / inductor-test / test (inductor_torchbench, 2, 2, linux.g5.4xlarge.nvidia.gpu) (gh)
Process completed with exit code 254.
inductor / unit-test / inductor-cpu-test / test (inductor_amx, 1, 2, linux.8xlarge.amx) (gh)
sccache: error: couldn't connect to server
inductor / unit-test / inductor-cpu-test / test (inductor_amx, 2, 2, linux.8xlarge.amx) (gh)
Process completed with exit code 254.
inductor / unit-test / inductor-cpu-test / test (inductor_avx2, 1, 2, linux.10xlarge.avx2) (gh)
sccache: error: couldn't connect to server
inductor / unit-test / inductor-test / test (inductor_cpp_wrapper, 1, 2, linux.g5.4xlarge.nvidia.gpu) (gh)
Process completed with exit code 254.
inductor / unit-test / inductor-test / test (inductor_cpp_wrapper, 2, 2, linux.g5.4xlarge.nvidia.gpu) (gh)
Process completed with exit code 254.
inductor / unit-test / inductor-test / test (inductor_distributed, 1, 1, linux.g5.12xlarge.nvidia.gpu) (gh)
Process completed with exit code 254.
inductor / unit-test / inductor-test / test (inductor, 1, 2, linux.g5.4xlarge.nvidia.gpu) (gh)
Process completed with exit code 254.
inductor / unit-test / inductor-test / test (inductor, 2, 2, linux.g5.4xlarge.nvidia.gpu) (gh)
Process completed with exit code 254.
inductor / unit-test / linux-jammy-cpu-py3.12-gcc11-inductor-triton-cpu / test (inductor-triton-cpu, 1, 1, linux.12xlarge) (gh)
Process completed with exit code 254.
Lint / lintrunner-noclang / linux-job (gh)
>>> Lint for test/distributed/test_inductor_collectives.py:
pull / cuda12.8-py3.10-gcc9-sm75 / test (pr_time_benchmarks, 1, 1, linux.g4dn.metal.nvidia.gpu) (gh)
Process completed with exit code 254.
pull / linux-docs / build-docs-cpp-false (gh)

CANCELLED JOBS - The following jobs were cancelled. Please retry:

This comment was automatically generated by Dr. CI and updates every 15 minutes.

ghstack-source-id: bc07003 Pull-Request: #164839

[ghstack-poisoned]

ghstack-source-id: edef03c Pull-Request: #164839

eellison · 2025-10-14T18:26:54Z

@pytorchbot rebase

pytorchmergebot · 2025-10-14T18:28:23Z

@pytorchbot started a rebase job onto refs/remotes/origin/viable/strict. Check the current status here

pytorchmergebot · 2025-10-14T18:28:30Z

Rebase failed due to Command git -C /home/runner/work/pytorch/pytorch rebase refs/remotes/origin/viable/strict gh/eellison/841/orig returned non-zero exit code 1

warning: skipped previously applied commit e03d4b5eafb
warning: skipped previously applied commit 3a66fa84b8b
warning: skipped previously applied commit 7069cf33c48
warning: skipped previously applied commit 617df7d0991
hint: use --reapply-cherry-picks to include skipped commits
hint: Disable this message with "git config set advice.skippedCherryPicks false"
Rebasing (1/2)
Auto-merging torch/_inductor/fx_passes/memory_estimator.py
CONFLICT (content): Merge conflict in torch/_inductor/fx_passes/memory_estimator.py
error: could not apply 2ec8337ba35... Refactor memory estimator to use node storages, add test
hint: Resolve all conflicts manually, mark them as resolved with
hint: "git add/rm <conflicted_files>", then run "git rebase --continue".
hint: You can instead skip this commit: run "git rebase --skip".
hint: To abort and get back to the state before "git rebase", run "git rebase --abort".
hint: Disable this message with "git config set advice.mergeConflict false"
Could not apply 2ec8337ba35... # Refactor memory estimator to use node storages, add test

Raised by https://github.com/pytorch/pytorch/actions/runs/18506301971

Update

3fd1f73

[ghstack-poisoned]

pytorch-bot bot added ciflow/inductor module: inductor oncall: distributed Add this issue/PR to distributed oncall triage queue labels Oct 7, 2025

eellison added a commit that referenced this pull request Oct 7, 2025

consider collective inputs only deallocated on wait()

675d21f

ghstack-source-id: bc07003 Pull-Request: #164839

eellison mentioned this pull request Oct 7, 2025

Refactor memory estimator to use node storages, add test #164783

Closed

eellison added the topic: not user facing topic category label Oct 7, 2025

eellison requested review from IvanKobzarev and ruisizhang123 October 7, 2025 15:49

Update

391f290

[ghstack-poisoned]

eellison added a commit that referenced this pull request Oct 7, 2025

consider collective inputs only deallocated on wait()

cb26d7f

ghstack-source-id: edef03c Pull-Request: #164839

IvanKobzarev approved these changes Oct 8, 2025

View reviewed changes

eellison closed this Oct 15, 2025

github-actions bot deleted the gh/eellison/841/head branch November 15, 2025 02:14

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

consider collective inputs only deallocated on wait() #164839

consider collective inputs only deallocated on wait() #164839

Uh oh!

eellison commented Oct 7, 2025 •

edited

Loading

Uh oh!

pytorch-bot bot commented Oct 7, 2025 •

edited

Loading

Uh oh!

eellison commented Oct 14, 2025

Uh oh!

pytorchmergebot commented Oct 14, 2025

Uh oh!

pytorchmergebot commented Oct 14, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

consider collective inputs only deallocated on wait() #164839

consider collective inputs only deallocated on wait() #164839

Uh oh!

Conversation

eellison commented Oct 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pytorch-bot bot commented Oct 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/164839

❗ 1 Active SEVs

❌ 21 New Failures, 34 Cancelled Jobs

Uh oh!

eellison commented Oct 14, 2025

Uh oh!

pytorchmergebot commented Oct 14, 2025

Uh oh!

pytorchmergebot commented Oct 14, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

eellison commented Oct 7, 2025 •

edited

Loading

pytorch-bot bot commented Oct 7, 2025 •

edited

Loading