Skip to content

[ROCm][CI] Create periodic-rocm-mi200.yml#166544

Closed
amdfaa wants to merge 8 commits intopytorch:mainfrom
amdfaa:patch-25
Closed

[ROCm][CI] Create periodic-rocm-mi200.yml#166544
amdfaa wants to merge 8 commits intopytorch:mainfrom
amdfaa:patch-25

Conversation

@amdfaa
Copy link
Contributor

@amdfaa amdfaa commented Oct 29, 2025 •
edited by jithunnair-amd
Loading

  • We are separating out the rocm jobs of the periodic workflow
  • We are introducing a new label ciflow/periodic-rocm-mi200 to allow us to run distributed tests only on ROCm runners, without triggering many other jobs on the periodic.yml workflow (via ciflow/periodic)
  • This new workflow will also be triggered via the ciflow/periodic, thus maintaining the old status quo.
  • We are reverting to the linux.rocm.gpu.4 label since it targets a lot more CI nodes at this point than the K8s/ARC-based linux.rocm.gpu.mi250.4 label, as that is still having some network/scaling issues.

cc @jeffdaily @sunway513 @jithunnair-amd @pruthvistony @ROCmSupport @dllehr-amd @jataylo @hongxiayang @naromero77amd

@amdfaa amdfaa requested a review from a team as a code owner October 29, 2025 17:34
@pytorch-bot
Copy link

pytorch-bot bot commented Oct 29, 2025 •
edited
Loading

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/166544

Note: Links to docs will display an error until the docs builds have been completed.

❗ 1 Active SEVs

There are 1 currently active SEVs. If your PR is affected, please view them below:

⏳ No Failures, 13 Pending

As of commit 98717aa with merge base deb7763 (image):
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@pytorch-bot pytorch-bot bot added the topic: not user facing topic category label Oct 29, 2025
@jeffdaily jeffdaily changed the title Create periodic-mi200 [ROCm][CI] Create periodic-mi200 Oct 29, 2025
@pytorch-bot pytorch-bot bot added the module: rocm AMD GPU support for Pytorch label Oct 29, 2025
jithunnair-amd
jithunnair-amd reviewed Oct 29, 2025
View reviewed changes
jithunnair-amd
jithunnair-amd reviewed Oct 29, 2025
View reviewed changes
pragupta
pragupta reviewed Oct 29, 2025
View reviewed changes
@jithunnair-amd jithunnair-amd changed the title [ROCm][CI] Create periodic-mi200 [ROCm][CI] Create periodic-rocm-mi200.yml Oct 29, 2025
@pytorch-bot pytorch-bot bot added the ciflow/rocm Trigger "default" config CI on ROCm label Oct 29, 2025
@pytorch-bot pytorch-bot bot removed the ciflow/rocm Trigger "default" config CI on ROCm label Oct 29, 2025
@jeffdaily
Copy link
Collaborator

jeffdaily commented Oct 29, 2025

@pytorchbot merge

@pytorch-bot pytorch-bot bot added the ciflow/trunk Trigger trunk jobs on your pull request label Oct 29, 2025
@pytorchmergebot
Copy link
Collaborator

pytorchmergebot commented Oct 29, 2025

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging
Check the merge workflow status
here

@jithunnair-amd jithunnair-amd added the ciflow/periodic-rocm-mi200 Trigger "distributed" config CI on ROCm MI200 label Oct 29, 2025
@pytorch-bot
Copy link

pytorch-bot bot commented Oct 29, 2025

Warning: Unknown label ciflow/periodic-rocm-mi200.
Currently recognized labels are

  • ciflow/b200
  • ciflow/b200-symm-mem
  • ciflow/b200-distributed
  • ciflow/binaries
  • ciflow/binaries_libtorch
  • ciflow/binaries_wheel
  • ciflow/h100
  • ciflow/h100-cutlass-backend
  • ciflow/h100-distributed
  • ciflow/h100-symm-mem
  • ciflow/inductor
  • ciflow/inductor-cu126
  • ciflow/inductor-micro-benchmark
  • ciflow/inductor-micro-benchmark-cpu-x86
  • ciflow/inductor-perf-compare
  • ciflow/inductor-perf-test-nightly-rocm-mi300
  • ciflow/inductor-perf-test-nightly-rocm-mi355
  • ciflow/inductor-perf-test-nightly-x86-zen
  • ciflow/inductor-periodic
  • ciflow/inductor-rocm
  • ciflow/linux-aarch64
  • ciflow/mps
  • ciflow/nightly
  • ciflow/op-benchmark
  • ciflow/periodic
  • ciflow/periodic-rocm-mi300
  • ciflow/pull
  • ciflow/quantization-periodic
  • ciflow/riscv64
  • ciflow/rocm
  • ciflow/rocm-mi300
  • ciflow/rocm-mi355
  • ciflow/rocm-navi31
  • ciflow/s390
  • ciflow/slow
  • ciflow/torchbench
  • ciflow/triton_binaries
  • ciflow/trunk
  • ciflow/unstable
  • ciflow/vllm
  • ciflow/win-arm64
  • ciflow/xpu

Please add the new label to .github/pytorch-probot.yml

@pytorch-bot pytorch-bot bot removed ciflow/trunk Trigger trunk jobs on your pull request ciflow/periodic-rocm-mi200 Trigger "distributed" config CI on ROCm MI200 labels Oct 30, 2025
@pytorchmergebot
Copy link
Collaborator

pytorchmergebot commented Oct 30, 2025

Merge failed

Reason: New commits were pushed while merging. Please rerun the merge command.

Details for Dev Infra team Raised by workflow job

@jithunnair-amd
Copy link
Collaborator

jithunnair-amd commented Oct 30, 2025

@pytorchbot merge -f "Force merging to relieve MI2xx queueing and provide separate workflow to target ROCm MI2xx distributed jobs"

@jithunnair-amd jithunnair-amd mentioned this pull request Oct 30, 2025
Closed
@pytorchmergebot
Copy link
Collaborator

pytorchmergebot commented Oct 30, 2025

Merge started

Your change will be merged immediately since you used the force (-f) flag, bypassing any CI checks (ETA: 1-5 minutes). Please use -f as last resort and instead consider -i/--ignore-current to continue the merge ignoring current failures. This will allow currently pending tests to finish and report signal before the merge.

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging
Check the merge workflow status
here

This was referenced Oct 30, 2025
Closed
Closed
BoyuanFeng pushed a commit that referenced this pull request Oct 31, 2025
@amdfaa @jeffdaily @BoyuanFeng
[ROCm][CI] Create periodic-rocm-mi200.yml (#166544)
e4201f1 
* We are separating out the rocm jobs of the periodic workflow
* We are introducing a new label `ciflow/periodic-rocm-mi200` to allow us to run distributed tests only on ROCm runners, without triggering many other jobs on the `periodic.yml` workflow (via `ciflow/periodic`)
* This new workflow will also be triggered via the `ciflow/periodic`, thus maintaining the old status quo.
* We are reverting to the `linux.rocm.gpu.4` label since it targets a lot more CI nodes at this point than the K8s/ARC-based `linux.rocm.gpu.mi250.4` label, as that is still having some network/scaling issues.

Pull Request resolved: #166544
Approved by: https://github.com/jeffdaily

Co-authored-by: Jeff Daily <jeff.daily@amd.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@jeffdaily jeffdaily jeffdaily approved these changes

@pragupta pragupta pragupta left review comments

@jithunnair-amd jithunnair-amd jithunnair-amd left review comments

Assignees

No one assigned

Labels

Merged module: rocm AMD GPU support for Pytorch open source topic: not user facing topic category

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@amdfaa @jeffdaily @pytorchmergebot @jithunnair-amd @pragupta @pytorchbot