Remove outdated flaky models and enable deterministic algorithms on ROCm by jataylo · Pull Request #169024 · pytorch/pytorch

jataylo · 2025-11-24T23:55:13Z

Remove outdated flaky models and enable deterministic algorithms on ROCm

cc @jeffdaily @sunway513 @jithunnair-amd @pruthvistony @ROCmSupport @hongxiayang @naromero77amd @pragupta @jerrymannil @xinyazhang @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @wenzhe-nrv @jiayisunx @chenyang78 @kadeng @chauhang @amjames @Lucaskabela

pytorch-bot · 2025-11-24T23:55:19Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/169024

📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓ Need help or want to give feedback on the CI? Visit the bot commands wiki

Note: Links to docs will display an error until the docs builds have been completed.

❌ 3 New Failures, 12 Unrelated Failures

As of commit b4c1548 with merge base 25c7201 ():

NEW FAILURES - The following jobs have failed:

inductor-periodic / rocm-periodic-dynamo-benchmarks-test / test (aot_inductor_huggingface, 1, 1, linux.rocm.gpu.gfx942.1) (gh)
RuntimeError: Found <class 'transformers.cache_utils.EncoderDecoderCache'> in output, which is not a known type. If this type holds tensors, you need to register a pytree for it. See https://github.com/pytorch/functorch/issues/475 for a brief explanation why. If you don't need to register a pytree, please leave a comment explaining your use case and we'll make this more ergonomic to deal with
inductor-periodic / rocm-periodic-dynamo-benchmarks-test / test (aot_inductor_torchbench, 2, 2, linux.rocm.gpu.gfx942.1) (gh)
tts_angular
inductor-periodic / rocm-periodic-dynamo-benchmarks-test / test (dynamic_aot_eager_huggingface, 1, 1, linux.rocm.gpu.gfx942.1) (gh)
GPT2ForSequenceClassification

FLAKY - The following jobs failed but were likely due to flakiness present on trunk:

inductor-perf-nightly-rocm-mi300 / rocm-py3_10-inductor-benchmark-test / test (inductor_torchbench_perf_rocm_mi300, 4, 9, linux.rocm.gpu.gfx942.1) (gh) (detected as infra flaky with no log or failing log classifier)
inductor-perf-nightly-rocm-mi355 / rocm-py3_10-inductor-benchmark-test / test (inductor_huggingface_perf_rocm_mi355, 5, 5, linux.rocm.gpu.mi355.1) (gh) (detected as infra flaky with no log or failing log classifier)
inductor-periodic / periodic-dynamo-benchmarks-test / test (aot_eager_huggingface, 1, 1, linux.g5.4xlarge.nvidia.gpu) (gh) (similar failure)
FAIL: graph_breaks=
inductor-periodic / periodic-dynamo-benchmarks-test / test (dynamic_aot_eager_huggingface, 1, 1, linux.g5.4xlarge.nvidia.gpu) (gh) (similar failure)
FAIL: graph_breaks=
inductor-periodic / periodic-dynamo-benchmarks-test / test (dynamic_inductor_huggingface, 1, 1, linux.g5.4xlarge.nvidia.gpu) (gh) (similar failure)
FAIL: graph_breaks=
inductor-periodic / periodic-dynamo-benchmarks-test / test (dynamo_eager_huggingface, 1, 1, linux.g5.4xlarge.nvidia.gpu) (gh) (similar failure)
FAIL: graph_breaks=
inductor-periodic / periodic-dynamo-benchmarks-test-cuda13 / test (aot_eager_huggingface, 1, 1, linux.g5.4xlarge.nvidia.gpu) (gh) (similar failure)
FAIL: graph_breaks=
inductor-periodic / periodic-dynamo-benchmarks-test-cuda13 / test (dynamic_aot_eager_huggingface, 1, 1, linux.g5.4xlarge.nvidia.gpu) (gh) (similar failure)
FAIL: graph_breaks=
inductor-periodic / periodic-dynamo-benchmarks-test-cuda13 / test (dynamic_inductor_huggingface, 1, 1, linux.g5.4xlarge.nvidia.gpu) (gh) (similar failure)
FAIL: graph_breaks=
inductor-periodic / periodic-dynamo-benchmarks-test-cuda13 / test (dynamo_eager_huggingface, 1, 1, linux.g5.4xlarge.nvidia.gpu) (gh) (similar failure)
FAIL: graph_breaks=
rocm-mi300 / linux-noble-rocm-py3.12-mi300 / test (default, 1, 6, linux.rocm.gpu.gfx942.1.b) (gh) (disabled by #169481)
test/dynamo/test_model_output.py::TestHFPretrained::test_pretrained

UNSTABLE - The following job is marked as unstable, possibly due to flakiness on trunk:

inductor-periodic / inductor-smoke-test / test (inductor_torchbench_smoketest_perf, 1, 1, linux.aws.a100, unstable) (gh) (#167613)
ModuleNotFoundError: No module named 'certifi'

This comment was automatically generated by Dr. CI and updates every 15 minutes.

benchmarks/dynamo/common.py

jataylo · 2025-11-28T11:36:09Z

I guess we'll need ROCm runners to come back for this...

jataylo · 2025-12-01T10:25:47Z

@pytorchbot rebase

pytorchmergebot · 2025-12-01T10:27:21Z

@pytorchbot started a rebase job onto refs/remotes/origin/viable/strict. Check the current status here

pytorchmergebot · 2025-12-01T10:27:23Z

Rebase failed due to Command git -C /home/runner/work/pytorch/pytorch push -f https://github.com/jataylo/pytorch.git pull/169024/head:flakiness-deter returned non-zero exit code 128

remote: The 'AMD' enterprise forbids access via a personal access tokens (classic) if the token's lifetime is greater than 366 days. Please adjust your token's lifetime at the following URL: https://github.com/settings/tokens/779664343
fatal: unable to access 'https://github.com/jataylo/pytorch.git/': The requested URL returned error: 403

Raised by https://github.com/pytorch/pytorch/actions/runs/19819362064

Co-authored-by: Jeff Daily <jeff.daily@amd.com>

jataylo · 2025-12-09T10:54:54Z

The tts_angular and gpt2sequence failures are unrelated, merging this. Rest addressed in #168073

jataylo · 2025-12-09T10:55:10Z

@pytorchbot merge

pytorchmergebot · 2025-12-09T10:57:20Z

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

pytorchmergebot · 2025-12-09T10:58:04Z

Merge failed

Reason: 14 jobs have failed, first few of them are: inductor-periodic / rocm-periodic-dynamo-benchmarks-test / test (dynamic_aot_eager_huggingface, 1, 1, linux.rocm.gpu.gfx942.1), inductor-periodic / rocm-periodic-dynamo-benchmarks-test / test (aot_inductor_huggingface, 1, 1, linux.rocm.gpu.gfx942.1), inductor-periodic / rocm-periodic-dynamo-benchmarks-test / test (aot_inductor_torchbench, 2, 2, linux.rocm.gpu.gfx942.1), rocm-mi300 / linux-noble-rocm-py3.12-mi300 / test (default, 1, 6, linux.rocm.gpu.gfx942.1.b), inductor-perf-nightly-rocm-mi355 / rocm-py3_10-inductor-benchmark-test / test (inductor_huggingface_perf_rocm_mi355, 5, 5, linux.rocm.gpu.mi355.1)

Details for Dev Infra team

Raised by workflow job

jataylo · 2025-12-09T13:22:26Z

@pytorchbot merge -f "ROCm unrelated failures in benchmark suite, these failures have snuck in and aren't caused by the PR"

pytorchmergebot · 2025-12-09T13:24:02Z

Merge started

Your change will be merged immediately since you used the force (-f) flag, bypassing any CI checks (ETA: 1-5 minutes). Please use -f as last resort and instead consider -i/--ignore-current to continue the merge ignoring current failures. This will allow currently pending tests to finish and report signal before the merge.

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

…OCm (#169024) Summary: Remove outdated flaky models and enable deterministic algorithms on ROCm X-link: pytorch/pytorch#169024 Approved by: https://github.com/jeffdaily Reviewed By: izaitsevfb Differential Revision: D88758770 fbshipit-source-id: 8ea515ea1e7e954212da1e3b1dda89c89b87e438 Co-authored-by: Jeff Daily <jeff.daily@amd.com>

…OCm (pytorch#169024) Remove outdated flaky models and enable deterministic algorithms on ROCm Pull Request resolved: pytorch#169024 Approved by: https://github.com/jeffdaily Co-authored-by: Jeff Daily <jeff.daily@amd.com>

jataylo added ciflow/inductor-periodic ciflow/inductor-perf-test-nightly-rocm-mi300 Trigger inductor perf tests on ROCm MI300 ciflow/inductor-perf-test-nightly-rocm-mi355 Trigger inductor perf tests on ROCm MI355 labels Nov 24, 2025

pytorch-bot bot added ciflow/inductor ciflow/rocm Trigger "default" config CI on ROCm module: dynamo module: rocm AMD GPU support for Pytorch labels Nov 24, 2025

pytorchbot added the open source label Nov 25, 2025

jeffdaily reviewed Nov 25, 2025

View reviewed changes

benchmarks/dynamo/common.py Outdated Show resolved Hide resolved

jataylo mentioned this pull request Nov 26, 2025

[ROCm] enabling compile should not cause signficiant accuracy loss on many important models #168935

Open

jataylo requested a review from jeffdaily November 28, 2025 11:35

jataylo marked this pull request as ready for review November 28, 2025 11:36

jataylo mentioned this pull request Dec 1, 2025

[Inductor Dashboard] Enable deterministic algorithms for all models on ROCm #156592

Closed

jeffdaily approved these changes Dec 2, 2025

View reviewed changes

jeffdaily added the topic: not user facing topic category label Dec 2, 2025

jataylo force-pushed the flakiness-deter branch from d145df7 to 55f6f59 Compare December 4, 2025 02:19

jataylo and others added 9 commits December 8, 2025 10:49

Remove outdated flaky models and enable deterministic algorithms on ROCm

d0fd556

Fix

3c1a44a

Add warn_only @jeffdaily

6773b00

Co-authored-by: Jeff Daily <jeff.daily@amd.com>

Add resnet models back to flakiness

1be0d11

Simplify

d998993

Add back some flaky models

3874a34

Fix csvs and rebase

31c1c81

Update accuracy status for torchrec_dlrm benchmark

ff014a4

Update tts_angular status from fail to pass

daf802b

jataylo and others added 6 commits December 8, 2025 10:49

Update accuracy status for torchrec_dlrm benchmark

ca85277

Update demucs status from fail to pass

6abf0c4

Update dynamic_inductor_torchbench_training.csv

86ab9b7

Update dynamo_eager_torchbench_inference.csv

6a1ef60

Update check_accuracy.py

70d184d

Fixes

b4c1548

jataylo force-pushed the flakiness-deter branch from 48c27d1 to b4c1548 Compare December 8, 2025 10:50

pytorch-bot bot added the ciflow/trunk Trigger trunk jobs on your pull request label Dec 9, 2025

pytorchmergebot added the merging label Dec 9, 2025

pytorchmergebot removed the merging label Dec 9, 2025

pytorchmergebot added the merging label Dec 9, 2025

pytorchmergebot closed this in b53d925 Dec 9, 2025

pytorchmergebot added Merged and removed merging labels Dec 9, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Remove outdated flaky models and enable deterministic algorithms on ROCm#169024

Remove outdated flaky models and enable deterministic algorithms on ROCm#169024
jataylo wants to merge 15 commits intopytorch:mainfrom
jataylo:flakiness-deter

jataylo commented Nov 24, 2025 •

edited

Loading

Uh oh!

pytorch-bot bot commented Nov 24, 2025 •

edited

Loading

Uh oh!

Uh oh!

jataylo commented Nov 28, 2025

Uh oh!

jataylo commented Dec 1, 2025

Uh oh!

pytorchmergebot commented Dec 1, 2025

Uh oh!

pytorchmergebot commented Dec 1, 2025

Uh oh!

jataylo commented Dec 9, 2025

Uh oh!

jataylo commented Dec 9, 2025

Uh oh!

pytorchmergebot commented Dec 9, 2025

Uh oh!

pytorchmergebot commented Dec 9, 2025

Uh oh!

jataylo commented Dec 9, 2025

Uh oh!

pytorchmergebot commented Dec 9, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

jataylo commented Nov 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pytorch-bot bot commented Nov 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/169024

❌ 3 New Failures, 12 Unrelated Failures

Uh oh!

Uh oh!

jataylo commented Nov 28, 2025

Uh oh!

jataylo commented Dec 1, 2025

Uh oh!

pytorchmergebot commented Dec 1, 2025

Uh oh!

pytorchmergebot commented Dec 1, 2025

Uh oh!

jataylo commented Dec 9, 2025

Uh oh!

jataylo commented Dec 9, 2025

Uh oh!

pytorchmergebot commented Dec 9, 2025

Merge started

Uh oh!

pytorchmergebot commented Dec 9, 2025

Merge failed

Uh oh!

jataylo commented Dec 9, 2025

Uh oh!

pytorchmergebot commented Dec 9, 2025

Merge started

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

jataylo commented Nov 24, 2025 •

edited

Loading

pytorch-bot bot commented Nov 24, 2025 •

edited

Loading