[https://nvbugs/5667454][test] Fix Test Case as Chunked Attention not Supported on sm_120 #9260

yufeiwu-nv · 2025-11-18T08:51:00Z

Summary by CodeRabbit

Tests
- Updated GPU compute capability filtering for performance tests to better target specific hardware configurations
- Added timeout specifications to critical test entries for improved reliability
- Refined test suite by removing obsolete entries and introducing new test scenarios with appropriate resource requirements

Description

The Blackwell support for chunked attention is for sm100 only, not for sm120. Since LLaMA4 has lower priority than other key models, dev don't plan to add new chunked attention kernels for it.

Modify test case accordingly.

Test Coverage

PR Checklist

Please review the following before submitting your PR:

PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.
PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.
Test cases are provided for new code paths (see test instructions)
Any new dependencies have been scanned for license and vulnerabilities
CODEOWNERS updated if ownership changes
Documentation updated as needed
Update tava architecture diagram if there is a significant design change in PR.
The reviewers assigned automatically/manually are appropriate for the PR.
Please check this after reviewing the above items as appropriate for this PR.

GitHub Bot Help

/bot [-h] ['run', 'kill', 'skip', 'reuse-pipeline'] ...

Provide a user friendly way for developers to interact with a Jenkins server.

Run /bot [-h|--help] to print this help message.

See details below for each supported subcommand.

Details

run [--reuse-test (optional)pipeline-id --disable-fail-fast --skip-test --stage-list "A10-PyTorch-1, xxx" --gpu-type "A30, H100_PCIe" --test-backend "pytorch, cpp" --add-multi-gpu-test --only-multi-gpu-test --disable-multi-gpu-test --post-merge --extra-stage "H100_PCIe-TensorRT-Post-Merge-1, xxx" --detailed-log --debug(experimental)]

Launch build/test pipelines. All previously running jobs will be killed.

--reuse-test (optional)pipeline-id (OPTIONAL) : Allow the new pipeline to reuse build artifacts and skip successful test stages from a specified pipeline or the last pipeline if no pipeline-id is indicated. If the Git commit ID has changed, this option will be always ignored. The DEFAULT behavior of the bot is to reuse build artifacts and successful test results from the last pipeline.

--disable-reuse-test (OPTIONAL) : Explicitly prevent the pipeline from reusing build artifacts and skipping successful test stages from a previous pipeline. Ensure that all builds and tests are run regardless of previous successes.

--disable-fail-fast (OPTIONAL) : Disable fail fast on build/tests/infra failures.

--skip-test (OPTIONAL) : Skip all test stages, but still run build stages, package stages and sanity check stages. Note: Does NOT update GitHub check status.

--stage-list "A10-PyTorch-1, xxx" (OPTIONAL) : Only run the specified test stages. Examples: "A10-PyTorch-1, xxx". Note: Does NOT update GitHub check status.

--gpu-type "A30, H100_PCIe" (OPTIONAL) : Only run the test stages on the specified GPU types. Examples: "A30, H100_PCIe". Note: Does NOT update GitHub check status.

--test-backend "pytorch, cpp" (OPTIONAL) : Skip test stages which don't match the specified backends. Only support [pytorch, cpp, tensorrt, triton]. Examples: "pytorch, cpp" (does not run test stages with tensorrt or triton backend). Note: Does NOT update GitHub pipeline status.

--only-multi-gpu-test (OPTIONAL) : Only run the multi-GPU tests. Note: Does NOT update GitHub check status.

--disable-multi-gpu-test (OPTIONAL) : Disable the multi-GPU tests. Note: Does NOT update GitHub check status.

--add-multi-gpu-test (OPTIONAL) : Force run the multi-GPU tests in addition to running L0 pre-merge pipeline.

--post-merge (OPTIONAL) : Run the L0 post-merge pipeline instead of the ordinary L0 pre-merge pipeline.

--extra-stage "H100_PCIe-TensorRT-Post-Merge-1, xxx" (OPTIONAL) : Run the ordinary L0 pre-merge pipeline and specified test stages. Examples: --extra-stage "H100_PCIe-TensorRT-Post-Merge-1, xxx".

--detailed-log (OPTIONAL) : Enable flushing out all logs to the Jenkins console. This will significantly increase the log volume and may slow down the job.

--debug (OPTIONAL) : Experimental feature. Enable access to the CI container for debugging purpose. Note: Specify exactly one stage in the stage-list parameter to access the appropriate container environment. Note: Does NOT update GitHub check status.

For guidance on mapping tests to stage names, see docs/source/reference/ci-overview.md
and the scripts/test_to_stage_mapping.py helper.

kill

kill

Kill all running builds associated with pull request.

skip

skip --comment COMMENT

Skip testing for latest commit on pull request. --comment "Reason for skipping build/test" is required. IMPORTANT NOTE: This is dangerous since lack of user care and validation can cause top of tree to break.

reuse-pipeline

reuse-pipeline

Reuse a previous pipeline to validate current commit. This action will also kill all currently running builds associated with the pull request. IMPORTANT NOTE: This is dangerous since lack of user care and validation can cause top of tree to break.

yufeiwu-nv · 2025-11-18T08:52:34Z

/bot run

coderabbitai · 2025-11-18T08:53:54Z

📝 Walkthrough

Walkthrough

Two performance test configuration files are updated to refine GPU selection and test filtering. Compute capability filters are added to gate tests for specific GPU architectures, GPU wildcards are replaced, test entries are removed or added, timeouts are introduced on existing tests, and a test scenario label is renamed.

Changes

Cohort / File(s)	Summary
Performance test configuration updates `tests/integration/test_lists/qa/llm_perf_core.yml`	Adds compute capability filters (gte: 9.0, lt: 12.0) to conditional blocks. Removes GPU wildcard filters (-h100, -h200, -h20*). Renames "rcca case" label to "chunked attention case". Maintains existing test ranges and entries.
Performance test sanity suite modifications `tests/integration/test_lists/qa/llm_perf_sanity.yml`	Removes two llama_v4_maverick_17b_128e_fp8 test entries. Adds TIMEOUT(100) markers to multiple existing tests. Introduces new conditional block with system_gpu_count (gte: 8) and compute_capability (gte: 9.0, lt: 12.0) filters containing a new llama_v4_maverick_17b_128e_instruct_fp8 test variant. Retains gpt_oss_20b_fp4 test entry.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~8 minutes

The changes are primarily declarative configuration updates with consistent patterns across both files (adding compute capability filters, introducing timeouts, test entry modifications)
Limited logic complexity; focus is on verifying test entry correctness and filter applicability
Minor attention to ensure GPU compute capability thresholds align with intended hardware targeting and test parameter validity

Pre-merge checks and finishing touches

✅ Passed checks (3 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly describes the main change: fixing test cases for chunked attention not being supported on sm_120, which aligns with the PR's objective.
Description check	✅ Passed	The description explains the technical issue (Blackwell chunked attention support only for sm100, not sm120) and the solution (modifying test cases), though Test Coverage section is left empty.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.

✨ Finishing touches

🧪 Generate unit tests (beta)

Create PR with unit tests
Post copyable unit tests in a comment

📜 Recent review details

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between ee941ac and 71ac004.

📒 Files selected for processing (2)

tests/integration/test_lists/qa/llm_perf_core.yml (2 hunks)
tests/integration/test_lists/qa/llm_perf_sanity.yml (1 hunks)

🧰 Additional context used

🧠 Learnings (7)

📓 Common learnings

Learnt from: pengbowang-nv
Repo: NVIDIA/TensorRT-LLM PR: 7192
File: tests/integration/test_lists/test-db/l0_dgx_b200.yml:56-72
Timestamp: 2025-08-26T09:49:04.956Z
Learning: In TensorRT-LLM test configuration files, the test scheduling system handles wildcard matching with special rules that prevent duplicate test execution even when the same tests appear in multiple yaml files with overlapping GPU wildcards (e.g., "*b200*" and "*gb200*").

Learnt from: nzmora-nvidia
Repo: NVIDIA/TensorRT-LLM PR: 9163
File: tensorrt_llm/_torch/auto_deploy/custom_ops/quant.py:107-113
Timestamp: 2025-11-14T11:22:03.729Z
Learning: In TensorRT-LLM AutoDeploy custom ops, when adding hardware capability checks to select between kernel implementations (e.g., cuBLAS vs. CUDA kernel), use descriptive variable names that identify the specific GPU architectures or families being targeted (e.g., `is_blackwell_geforce_or_ada`) rather than generic names like `enable_cuda_core`. This makes it clear that the code is selecting an implementation path based on hardware capabilities, not enabling/disabling hardware features.

Learnt from: fredricz-20070104
Repo: NVIDIA/TensorRT-LLM PR: 7645
File: tests/integration/test_lists/qa/llm_function_core.txt:648-648
Timestamp: 2025-09-09T09:40:45.658Z
Learning: In TensorRT-LLM test lists, it's common and intentional for the same test to appear in multiple test list files when they serve different purposes (e.g., llm_function_core.txt for comprehensive core functionality testing and llm_function_core_sanity.txt for quick sanity checks). This duplication allows tests to be run in different testing contexts.

📚 Learning: 2025-08-26T09:49:04.956Z

Learnt from: pengbowang-nv
Repo: NVIDIA/TensorRT-LLM PR: 7192
File: tests/integration/test_lists/test-db/l0_dgx_b200.yml:56-72
Timestamp: 2025-08-26T09:49:04.956Z
Learning: In TensorRT-LLM test configuration files, the test scheduling system handles wildcard matching with special rules that prevent duplicate test execution even when the same tests appear in multiple yaml files with overlapping GPU wildcards (e.g., "*b200*" and "*gb200*").

Applied to files:

tests/integration/test_lists/qa/llm_perf_core.yml
tests/integration/test_lists/qa/llm_perf_sanity.yml

📚 Learning: 2025-09-09T09:40:45.658Z

Learnt from: fredricz-20070104
Repo: NVIDIA/TensorRT-LLM PR: 7645
File: tests/integration/test_lists/qa/llm_function_core.txt:648-648
Timestamp: 2025-09-09T09:40:45.658Z
Learning: In TensorRT-LLM test lists, it's common and intentional for the same test to appear in multiple test list files when they serve different purposes (e.g., llm_function_core.txt for comprehensive core functionality testing and llm_function_core_sanity.txt for quick sanity checks). This duplication allows tests to be run in different testing contexts.

Applied to files:

tests/integration/test_lists/qa/llm_perf_core.yml
tests/integration/test_lists/qa/llm_perf_sanity.yml

📚 Learning: 2025-11-14T11:22:03.729Z

Learnt from: nzmora-nvidia
Repo: NVIDIA/TensorRT-LLM PR: 9163
File: tensorrt_llm/_torch/auto_deploy/custom_ops/quant.py:107-113
Timestamp: 2025-11-14T11:22:03.729Z
Learning: In TensorRT-LLM AutoDeploy custom ops, when adding hardware capability checks to select between kernel implementations (e.g., cuBLAS vs. CUDA kernel), use descriptive variable names that identify the specific GPU architectures or families being targeted (e.g., `is_blackwell_geforce_or_ada`) rather than generic names like `enable_cuda_core`. This makes it clear that the code is selecting an implementation path based on hardware capabilities, not enabling/disabling hardware features.

Applied to files:

tests/integration/test_lists/qa/llm_perf_core.yml

📚 Learning: 2025-09-17T02:48:52.732Z

Learnt from: tongyuantongyu
Repo: NVIDIA/TensorRT-LLM PR: 7781
File: tests/integration/test_lists/waives.txt:313-313
Timestamp: 2025-09-17T02:48:52.732Z
Learning: In TensorRT-LLM, `tests/integration/test_lists/waives.txt` is specifically for waiving/skipping tests, while other test list files like those in `test-db/` and `qa/` directories are for different test execution contexts (pre-merge, post-merge, QA tests). The same test appearing in both waives.txt and execution list files is intentional - the test is part of test suites but will be skipped due to the waiver.

Applied to files:

tests/integration/test_lists/qa/llm_perf_core.yml
tests/integration/test_lists/qa/llm_perf_sanity.yml

📚 Learning: 2025-07-28T17:06:08.621Z

Learnt from: moraxu
Repo: NVIDIA/TensorRT-LLM PR: 6303
File: tests/integration/test_lists/qa/examples_test_list.txt:494-494
Timestamp: 2025-07-28T17:06:08.621Z
Learning: In TensorRT-LLM testing, it's common to have both CLI flow tests (test_cli_flow.py) and PyTorch API tests (test_llm_api_pytorch.py) for the same model. These serve different purposes: CLI flow tests validate the traditional command-line workflow, while PyTorch API tests validate the newer LLM API backend. Both are legitimate and should coexist.

Applied to files:

tests/integration/test_lists/qa/llm_perf_core.yml
tests/integration/test_lists/qa/llm_perf_sanity.yml

📚 Learning: 2025-08-13T11:07:11.772Z

Learnt from: Funatiq
Repo: NVIDIA/TensorRT-LLM PR: 6754
File: tests/integration/test_lists/test-db/l0_a30.yml:41-47
Timestamp: 2025-08-13T11:07:11.772Z
Learning: In TensorRT-LLM test configuration files like tests/integration/test_lists/test-db/l0_a30.yml, TIMEOUT values are specified in minutes, not seconds.

Applied to files:

tests/integration/test_lists/qa/llm_perf_sanity.yml

⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (1)

GitHub Check: Pre-commit Check

🔇 Additional comments (5)

tests/integration/test_lists/qa/llm_perf_core.yml (2)

275-284: Compute capability filter correctly gates chunked attention tests to Hopper and earlier architectures.

The new compute_capability: {gte: 9.0, lt: 12.0} filter (lines 277–279) appropriately excludes Blackwell (compute capability 12.0+) from chunked attention test execution, which aligns with the PR objective that chunked attention kernels are only supported for sm_100 and not sm_120.

311-311: Improved test scenario label for clarity.

The comment update from the inferred "rcca case" to explicit "chunked attention case" on line 311 improves code maintainability and makes the test intent clearer.

tests/integration/test_lists/qa/llm_perf_sanity.yml (3)

171-173: TIMEOUT markers added to performance tests.

TIMEOUT(100) and TIMEOUT(60) markers have been added to three tests, which is appropriate test maintenance for performance workloads that may have longer execution times on specific hardware.

179-190: New conditional block mirrors core file's compute_capability gating for chunked attention tests.

The new conditional block (lines 179–187) with compute_capability: {gte: 9.0, lt: 12.0} correctly gates the chunked attention test case (line 189–190) to Hopper and earlier, excluding Blackwell. This is consistent with the matching gating applied in llm_perf_core.yml and aligns with the PR's objective to exclude sm_120 from chunked attention test execution.

The test parameters on line 190 (kv_frac: 0.6, maxnt: 20000) indicate a chunked attention workload, which is appropriately guarded by the compute capability filter. Per the learnings, test duplication across core and sanity contexts is intentional and expected.

179-179: Clarify comment intent regarding RTX 6000 exclusion.

The comment on line 179 states "exclude GB20X, RTX 6000 for not supported," but RTX 6000 is not Blackwell (GB20X) and uses a separate wildcard-based condition block elsewhere in this file (see lines 370–408). Clarify whether this comment intends to reference a different GPU family or architecture, or if it can be simplified to focus solely on Blackwell exclusion.

Tip

📝 Customizable high-level summaries are now available in beta!

You can now customize how CodeRabbit generates the high-level summary in your pull requests — including its content, structure, tone, and formatting.

Provide your own instructions using the high_level_summary_instructions setting.
Format the summary however you like (bullet lists, tables, multi-section layouts, contributor stats, etc.).
Use high_level_summary_in_walkthrough to move the summary from the description to the walkthrough section.

Example instruction:

"Divide the high-level summary into five sections:

📝 Description — Summarize the main change in 50–60 words, explaining what was done.

📓 References — List relevant issues, discussions, documentation, or related PRs.

📦 Dependencies & Requirements — Mention any new/updated dependencies, environment variable changes, or configuration updates.

📊 Contributor Summary — Include a Markdown table showing contributions:
| Contributor | Lines Added | Lines Removed | Files Changed |

✔️ Additional Notes — Add any extra reviewer context.
Keep each section concise (under 200 words) and use bullet or numbered lists for clarity."

Note: This feature is currently in beta for Pro-tier users, and pricing will be announced later.

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

tensorrt-cicd · 2025-11-18T08:59:29Z

PR_Github #24885 [ run ] triggered by Bot. Commit: 244b929

Signed-off-by: yufeiwu-nv <230315618+yufeiwu-nv@users.noreply.github.com>

tensorrt-cicd · 2025-11-18T09:48:52Z

PR_Github #24885 [ run ] completed with state FAILURE. Commit: 244b929
/LLM/main/L0_MergeRequest_PR pipeline #18790 completed with status: 'FAILURE'

yufeiwu-nv · 2025-11-19T05:45:03Z

/bot run

tensorrt-cicd · 2025-11-19T05:51:07Z

PR_Github #24989 [ run ] triggered by Bot. Commit: 408ca8a

yufeiwu-nv · 2025-11-19T08:47:44Z

/bot run

tensorrt-cicd · 2025-11-19T08:53:18Z

PR_Github #25028 [ run ] triggered by Bot. Commit: 71ac004

tensorrt-cicd · 2025-11-19T13:22:37Z

PR_Github #25028 [ run ] completed with state SUCCESS. Commit: 71ac004
/LLM/main/L0_MergeRequest_PR pipeline #18910 completed with status: 'FAILURE'

yufeiwu-nv · 2025-11-20T08:15:37Z

/bot skip --comment "only test list modify"

tensorrt-cicd · 2025-11-20T08:21:34Z

PR_Github #25177 [ skip ] triggered by Bot. Commit: fc4dadb

tensorrt-cicd · 2025-11-20T08:58:40Z

PR_Github #25177 [ skip ] completed with state SUCCESS. Commit: fc4dadb
Skipping testing for commit fc4dadb

yufeiwu-nv requested review from LarryXFly, StanleySun639 and ruodil November 18, 2025 08:51

yufeiwu-nv changed the title ~~[https://nvbugs/5667454 ][test] Fix Test Case as Chunked Attention not Supported on sm_120~~ [https://nvbugs/5667454][test] Fix Test Case as Chunked Attention not Supported on sm_120 Nov 18, 2025

StanleySun639 approved these changes Nov 18, 2025

View reviewed changes

yufeiwu-nv force-pushed the modify_chunked_attention_test branch from a86a92f to 0c670ee Compare November 18, 2025 08:56

yufeiwu-nv requested a review from a team as a code owner November 18, 2025 08:56

yufeiwu-nv requested review from QiJune and kaiyux November 18, 2025 08:56

yufeiwu-nv force-pushed the modify_chunked_attention_test branch 4 times, most recently from 3827db3 to 4da2fe2 Compare November 18, 2025 09:25

Fix test case as chunked attention not Supported on sm_120

4da2fe2

Signed-off-by: yufeiwu-nv <230315618+yufeiwu-nv@users.noreply.github.com>

yufeiwu-nv removed request for QiJune and kaiyux November 18, 2025 09:25

fix pre-commit issue

408ca8a

Signed-off-by: yufeiwu-nv <230315618+yufeiwu-nv@users.noreply.github.com>

Merge branch 'main' into modify_chunked_attention_test

71ac004

Merge branch 'main' into modify_chunked_attention_test

fc4dadb

yufeiwu-nv enabled auto-merge (squash) November 20, 2025 08:15

yufeiwu-nv merged commit 0e746fa into NVIDIA:main Nov 20, 2025
5 checks passed

[https://nvbugs/5667454][test] Fix Test Case as Chunked Attention not Supported on sm_120 #9260

[https://nvbugs/5667454][test] Fix Test Case as Chunked Attention not Supported on sm_120 #9260

Uh oh!

Conversation

yufeiwu-nv commented Nov 18, 2025 • edited by coderabbitai bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary by CodeRabbit

Description

Test Coverage

PR Checklist

GitHub Bot Help

kill

skip

reuse-pipeline

Uh oh!

yufeiwu-nv commented Nov 18, 2025

Uh oh!

coderabbitai bot commented Nov 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Estimated code review effort

Pre-merge checks and finishing touches

Uh oh!

tensorrt-cicd commented Nov 18, 2025

Uh oh!

tensorrt-cicd commented Nov 18, 2025

Uh oh!

yufeiwu-nv commented Nov 19, 2025

Uh oh!

tensorrt-cicd commented Nov 19, 2025

Uh oh!

yufeiwu-nv commented Nov 19, 2025

Uh oh!

tensorrt-cicd commented Nov 19, 2025

Uh oh!

tensorrt-cicd commented Nov 19, 2025

Uh oh!

yufeiwu-nv commented Nov 20, 2025

Uh oh!

tensorrt-cicd commented Nov 20, 2025

Uh oh!

tensorrt-cicd commented Nov 20, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

yufeiwu-nv commented Nov 18, 2025 •

edited by coderabbitai bot

Loading

coderabbitai bot commented Nov 18, 2025 •

edited

Loading