[ROCm][inductor] heuristic improvements for reduction kernels by naromero77amd · Pull Request #161280 · pytorch/pytorch

naromero77amd · 2025-08-22T16:59:22Z

Improvements to reduction kernel heuristics for MI350.

Contributions from several members of the AMD Inductor and Triton teams: @jataylo @iupaikov-amd @AmdSampsa @xiaohuguo2023

cc @jeffdaily @sunway513 @jithunnair-amd @pruthvistony @ROCmSupport @jataylo @hongxiayang @pragupta @jerrymannil @xinyazhang @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @wenzhe-nrv @jiayisunx @ipiszy @kadeng @muchulee8 @amjames @chauhang @aakhundov @coconutruben @mlazos @dllehr-amd @chenyang78

pytorch-bot · 2025-08-22T16:59:26Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/161280

📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓ Need help or want to give feedback on the CI? Visit the bot commands wiki

Note: Links to docs will display an error until the docs builds have been completed.

❌ 1 New Failure, 1 Unrelated Failure

As of commit bb52a1d with merge base d74f9ec ():

NEW FAILURE - The following job has failed:

trunk / linux-jammy-rocm-py3.10 / build (gh)
ninja: build stopped: subcommand failed

FLAKY - The following job failed but was likely due to flakiness present on trunk:

inductor / unit-test / inductor-pallas-cpu-test / test (inductor-pallas-cpu, 1, 1, linux.12xlarge) (gh) (similar failure)
Process completed with exit code 1.

This comment was automatically generated by Dr. CI and updates every 15 minutes.

pytorch-bot · 2025-08-22T16:59:37Z

To add the ciflow label ciflow/inductor please first approve the workflows that are awaiting approval (scroll to the bottom of this page).

This helps ensure we don't trigger CI on this PR until it is actually authorized to do so. Please ping one of the reviewers if you do not have access to approve and run workflows.

pytorch-bot · 2025-08-22T16:59:37Z

To add the ciflow label ciflow/rocm please first approve the workflows that are awaiting approval (scroll to the bottom of this page).

This helps ensure we don't trigger CI on this PR until it is actually authorized to do so. Please ping one of the reviewers if you do not have access to approve and run workflows.

facebook-github-bot · 2025-08-22T20:15:31Z

@haoyuz has imported this pull request. If you are a Meta employee, you can view this in D80836468.

facebook-github-bot · 2025-08-25T22:34:25Z

@haoyuz has imported this pull request. If you are a Meta employee, you can view this in D80836468.

facebook-github-bot · 2025-09-12T17:30:26Z

@haoyuz has imported this pull request. If you are a Meta employee, you can view this in D80836468.

jataylo · 2025-09-23T13:41:50Z

@pytorchbot rebase

pytorchmergebot · 2025-09-23T13:43:26Z

@pytorchbot started a rebase job onto refs/remotes/origin/viable/strict. Check the current status here

pytorchmergebot · 2025-09-23T13:43:28Z

Rebase failed due to Command git -C /home/runner/work/pytorch/pytorch rebase refs/remotes/origin/viable/strict pull/161280/head returned non-zero exit code 1

Rebasing (1/18)
Auto-merging torch/_inductor/codegen/triton.py
Auto-merging torch/_inductor/runtime/triton_heuristics.py
CONFLICT (content): Merge conflict in torch/_inductor/runtime/triton_heuristics.py
error: could not apply d1e5254957a... Added triton perf improvement changes
hint: Resolve all conflicts manually, mark them as resolved with
hint: "git add/rm <conflicted_files>", then run "git rebase --continue".
hint: You can instead skip this commit: run "git rebase --skip".
hint: To abort and get back to the state before "git rebase", run "git rebase --abort".
hint: Disable this message with "git config set advice.mergeConflict false"
Could not apply d1e5254957a... # Added triton perf improvement changes

Raised by https://github.com/pytorch/pytorch/actions/runs/17948121352

naromero77amd · 2025-12-19T18:54:41Z

Resolved conflict and will try to merge.

naromero77amd · 2025-12-19T18:54:57Z

@pytorchbot merge

naromero77amd · 2025-12-20T15:55:07Z

@pytorchbot rebase

@jataylo

Improvements to reduction kernel heuristics for MI350. Contributions from several members of the AMD Inductor and Triton teams: @jataylo @iupaikov-amd @AmdSampsa @xiaohuguo2023 **Duplicate of this PR:** #161280 (which has already been approved multiple times, but we are unable to merge due to some Meta Internal Check that cannot be cleared). Pull Request resolved: #170931 Approved by: https://github.com/jeffdaily

naromero77amd · 2025-12-21T15:17:03Z

Duplicate PR here was landed: #170931

FWIW, I think the real issue might have been the pytorchbot's token not confirming to AMD security policy.

@jataylo

Improvements to reduction kernel heuristics for MI350. Contributions from several members of the AMD Inductor and Triton teams: @jataylo @iupaikov-amd @AmdSampsa @xiaohuguo2023 **Duplicate of this PR:** #161280 (which has already been approved multiple times, but we are unable to merge due to some Meta Internal Check that cannot be cleared). Pull Request resolved: #170931 Approved by: https://github.com/jeffdaily

@jataylo

…h#170931) Improvements to reduction kernel heuristics for MI350. Contributions from several members of the AMD Inductor and Triton teams: @jataylo @iupaikov-amd @AmdSampsa @xiaohuguo2023 **Duplicate of this PR:** pytorch#161280 (which has already been approved multiple times, but we are unable to merge due to some Meta Internal Check that cannot be cleared). Pull Request resolved: pytorch#170931 Approved by: https://github.com/jeffdaily

@jataylo

…h#170931) Improvements to reduction kernel heuristics for MI350. Contributions from several members of the AMD Inductor and Triton teams: @jataylo @iupaikov-amd @AmdSampsa @xiaohuguo2023 **Duplicate of this PR:** pytorch#161280 (which has already been approved multiple times, but we are unable to merge due to some Meta Internal Check that cannot be cleared). Pull Request resolved: pytorch#170931 Approved by: https://github.com/jeffdaily (cherry picked from commit 5eceb87)

pytorch-bot bot added module: inductor module: rocm AMD GPU support for Pytorch labels Aug 22, 2025

naromero77amd marked this pull request as draft August 22, 2025 16:59

pytorch-bot bot added ciflow/inductor ciflow/rocm Trigger "default" config CI on ROCm labels Aug 22, 2025

pytorch-bot bot removed ciflow/inductor ciflow/rocm Trigger "default" config CI on ROCm labels Aug 22, 2025

pytorchbot added the open source label Aug 22, 2025

naromero77amd added the release notes: rocm mandatorylabel label Aug 22, 2025

naromero77amd force-pushed the mi350_perf_tuning branch from d08dd22 to 9b6ed63 Compare September 8, 2025 21:58

naromero77amd changed the title ~~[WIP][inductor][ROCm] MI350 perf tuning~~ [WIP][inductor][ROCm] MI350 reduction heuristics improvements Sep 11, 2025

naromero77amd marked this pull request as ready for review September 11, 2025 22:24

naromero77amd marked this pull request as draft September 11, 2025 22:24

naromero77amd changed the title ~~[WIP][inductor][ROCm] MI350 reduction heuristics improvements~~ [inductor][ROCm] MI350 reduction heuristics improvements Sep 26, 2025

pytorch-bot bot added ciflow/inductor ciflow/rocm Trigger "default" config CI on ROCm labels Sep 26, 2025

naromero77amd changed the title ~~[inductor][ROCm] MI350 reduction heuristics improvements~~ [ROCm][inductor] MI350 reduction heuristics improvements Sep 26, 2025

naromero77amd force-pushed the mi350_perf_tuning branch from 645f758 to 4eda531 Compare September 27, 2025 01:29

pytorch-bot bot removed ciflow/inductor ciflow/rocm Trigger "default" config CI on ROCm labels Sep 27, 2025

jeffdaily added the ciflow/inductor label Sep 27, 2025

naromero77amd added ciflow/inductor-rocm Trigger "inductor" config CI on ROCm ciflow/inductor ciflow/rocm Trigger "default" config CI on ROCm and removed ciflow/trunk Trigger trunk jobs on your pull request ciflow/inductor ciflow/inductor-rocm Trigger "inductor" config CI on ROCm labels Dec 16, 2025

naromero77amd force-pushed the mi350_perf_tuning branch from 3fd021b to 1eb723a Compare December 16, 2025 02:35

pytorch-bot bot added the ciflow/rocm-mi300 Trigger "default" config CI on ROCm MI300 label Dec 16, 2025

mlazos approved these changes Dec 17, 2025

View reviewed changes

naromero77amd added 8 commits December 19, 2025 18:48

Reduction heursitics improvements for ROCm

f64ffee

Add PropagateNan argument to minimum and maximum function.

630d930

Update autotune_lookup_tunable UT.

03094c0

Another fix to control flow.

576bff0

Fix UT that looks for specific Triton code string.

cbe6cb8

Fix UT that looks for specific Triton code. This time on NVIDIA.

3586e1c

Conditionalize pipelining on HIP and Triton version.

9397605

Lint

bb52a1d

naromero77amd force-pushed the mi350_perf_tuning branch from 1eb723a to bb52a1d Compare December 19, 2025 18:53

pytorch-bot bot added the ciflow/trunk Trigger trunk jobs on your pull request label Dec 19, 2025

naromero77amd mentioned this pull request Dec 20, 2025

[ROCm][inductor] heuristic improvements for reduction kernels #170931

Closed

naromero77amd closed this Dec 21, 2025

Conversation

naromero77amd commented Aug 22, 2025 • edited by pytorch-bot bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pytorch-bot bot commented Aug 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/161280

❌ 1 New Failure, 1 Unrelated Failure

Uh oh!

pytorch-bot bot commented Aug 22, 2025

Uh oh!

pytorch-bot bot commented Aug 22, 2025

Uh oh!

facebook-github-bot commented Aug 22, 2025

Uh oh!

facebook-github-bot commented Aug 25, 2025

Uh oh!

facebook-github-bot commented Sep 12, 2025

Uh oh!

jataylo commented Sep 23, 2025

Uh oh!

pytorchmergebot commented Sep 23, 2025

Uh oh!

pytorchmergebot commented Sep 23, 2025

Uh oh!

naromero77amd commented Dec 19, 2025

Uh oh!

naromero77amd commented Dec 19, 2025

Uh oh!

naromero77amd commented Dec 20, 2025

Uh oh!

naromero77amd commented Dec 21, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

13 participants

naromero77amd commented Aug 22, 2025 •

edited by pytorch-bot bot

Loading

pytorch-bot bot commented Aug 22, 2025 •

edited

Loading