Skip to content

fix : skip FP8 MHA accuracy checks on TRT-RTX 1.7 and earlier - #4925

Open
tp5uiuc wants to merge 2 commits into
pytorch:mainfrom
tp5uiuc:test/rtx-fp8-mha-version-skip
Open

fix : skip FP8 MHA accuracy checks on TRT-RTX 1.7 and earlier#4925
tp5uiuc wants to merge 2 commits into
pytorch:mainfrom
tp5uiuc:test/rtx-fp8-mha-version-skip

Conversation

@tp5uiuc

@tp5uiuc tp5uiuc commented Oct 23, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

Description

This PR adds a version skip for test_fp8_mha_fused_kernel on TensorRT-RTX 1.7 and earlier.
These versions return incorrect outputs for this case.
TensorRT-RTX 1.8 and later run the test.

Type of change

  • Bug fix (non-breaking change which fixes an issue)

Testing

  • The version utility checks pass for TRT-RTX 1.6, 1.7, 1.8, and 2.0, and standard TensorRT 11.4.
  • The L40S results below precede the utility change. The GPU allocation expired before a repeat run.
  • L40S, TRT-RTX 1.7.1.116: the test skips.
  • L40S, TRT-RTX 1.8.0.38: the test runs and reaches the known cosine failure, 0.625796.
  • All applicable pre-commit checks pass.

Checklist:

  • My code follows the style guidelines of this project
  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have added tests to verify my fix or my feature
  • New and existing unit tests pass locally with my changes
  • I have added the relevant labels to my PR

@tp5uiuc @codex
Skip FP8 MHA accuracy checks on TRT-RTX 1.7 and earlier
7ab06ca 
These versions return incorrect outputs for the fused attention case.

Co-Authored-By: Codex <noreply@openai.com>
@tp5uiuc tp5uiuc self-assigned this Oct 23, 2026
@meta-cla meta-cla Bot added the cla signed label Oct 23, 2026
@github-actions github-actions Bot added the component: tests Issues re: Tests label Oct 23, 2026
@tp5uiuc @codex
Use the version utility for the FP8 MHA test skip
8e557e8 
The shared utility defines the TensorRT-RTX version requirement consistently.

Co-Authored-By: Codex <noreply@openai.com>
@tp5uiuc tp5uiuc changed the title Skip FP8 MHA accuracy checks on TRT-RTX 1.7 and earlier fix : skip FP8 MHA accuracy checks on TRT-RTX 1.7 and earlier Oct 23, 2026
@tp5uiuc

tp5uiuc commented Oct 24, 2026

Copy link
Copy Markdown
Collaborator Author

[by Codex]

The three failure groups in the full CI run also occur on this PR's base commit, b0d9a1eb54, on main.

Failure PR jobs Evidence on main
TestEngineSetupMemory::test_setup_does_not_copy_the_plan Three Linux jobs: standard runtime, RTX runtime, and Python-only RTX runtime The same test fails. The memory assertion passes; the subsequent numeric comparison exceeds the tolerance.
test_dynamic_head_dim_with_hf_model[False-dtype0] Linux RTX critical models The same accuracy failure occurs: three mismatched values, with maximum absolute difference 0.4048 against 0.3 allowed.
Nine tests in test_turing_capability_guards.py Linux and Windows RTX models The same nine tests fail. Eight execute SM 7.5 engines on SM 8.6. One requests unsupported FP32 3D deconvolution on Turing.

The Linux model results record test_fp8_mha_fused_kernel as skipped because the GPU lacks compute capability 8.9.
It does not cause these failures.
The latest full CI runs for SBSA and Windows ARM64 pass.

These failures predate this PR. I found no regression from the FP8 version check.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@lanluo-nvidia lanluo-nvidia Awaiting requested review from lanluo-nvidia

Assignees

@tp5uiuc tp5uiuc

Labels

ci: full cla signed component: tests Issues re: Tests

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@tp5uiuc