[None][doc] Add doc for torch.compile & piecewise cuda graph #8527
Conversation
📝 WalkthroughWalkthroughThe pull request expands the pre-commit codespell hook configuration to skip SVG files in addition to existing ATTRIBUTIONS-*.md files, and adds a comprehensive documentation page explaining how to enable and use Torch Compile with Piecewise CUDA Graph in TensorRT LLM. Changes
Estimated code review effort🎯 2 (Simple) | ⏱️ ~15 minutes Pre-merge checks and finishing touches❌ Failed checks (1 warning)
✅ Passed checks (2 passed)
✨ Finishing touches
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 6
📜 Review details
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
⛔ Files ignored due to path filters (3)
docs/source/media/current_model_definition_ds.svgis excluded by!**/*.svgdocs/source/media/custom_backend_overview.svgis excluded by!**/*.svgdocs/source/media/piecewise_runner.pngis excluded by!**/*.png
📒 Files selected for processing (2)
.pre-commit-config.yaml(1 hunks)docs/source/torch/features/torch_compile_and_piecewise_cuda_graph.md(1 hunks)
🧰 Additional context used
🪛 LanguageTool
docs/source/torch/features/torch_compile_and_piecewise_cuda_graph.md
[grammar] ~79-~79: Use a hyphen to join words.
Context: ... for allreduce & rms norm. 1. AllReduce related fusion: Fuse the following opera...
(QB_NEW_EN_HYPHEN)
[grammar] ~105-~105: Use a hyphen to join words.
Context: ...d by user config 4. Insert multi-stream related custom op: since the Fx graph ex...
(QB_NEW_EN_HYPHEN)
[grammar] ~173-~173: Ensure spelling is correct
Context: ... so that it can be easily recognized by piesewise CUDA graph 4. Use some operators that w...
(QB_NEW_EN_ORTHOGRAPHY_ERROR_IDS_1)
[style] ~174-~174: To elevate your writing, try using a synonym here.
Context: ...ors that will cause graph break and are hard to avoid Reasons to wrap MoE into a la...
(HARD_TO)
[grammar] ~178-~178: Ensure spelling is correct
Context: ...nto a large custom op: 1. Use a lot of deepep ops that didn’t wrap into custom ops 2....
(QB_NEW_EN_ORTHOGRAPHY_ERROR_IDS_1)
[grammar] ~201-~201: Use a hyphen to join words.
Context: ...race failure, but it will introduce long time tracing that generally not acceptab...
(QB_NEW_EN_HYPHEN)
[style] ~202-~202: You have already used ‘try to’ in nearby sentences. Consider using one of the replacements to let your writing stand out.
Context: ...easily written into a custom op format, try to replace it with a custom op 2. If th...
(REP_TRY)
🪛 markdownlint-cli2 (0.18.1)
docs/source/torch/features/torch_compile_and_piecewise_cuda_graph.md
107-107: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
282-282: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (1)
- GitHub Check: Pre-commit Check
🔇 Additional comments (3)
.pre-commit-config.yaml (1)
69-69: LGTM!Adding SVG files to the codespell skip list is appropriate, as SVG markup can trigger false spell-check positives. This aligns with the new documentation media being added to the repository.
docs/source/torch/features/torch_compile_and_piecewise_cuda_graph.md (2)
1-300: Documentation is well-structured, technically sound, and all media assets are present.This documentation provides excellent guidance on enabling and using Torch Compile with Piecewise CUDA Graph in TensorRT LLM. The content covers clear usage instructions, in-depth technical background on the IR optimization pipeline, practical development guidelines, common failure modes and solutions, and recompilation considerations. All referenced media files have been verified to exist at the correct paths.
105-105: Fix redundant phrasing and ensure hyphenation.Line 105 contains "user user-defined" which is redundant. Also add hyphenation per LanguageTool feedback.
Apply this diff:
-Currently torch.compile won't create a subgraph for user user-defined CUDA stream. Instead it will convert it to `set_stream`. The set_stream op doesn't have any consumers, so it will be removed in the Torch IR to ATen IR transformation, thus losing all the multi-stream scheduling. +Currently torch.compile won't create a subgraph for user-defined CUDA streams. Instead it will convert it to `set_stream`. The set_stream op doesn't have any consumers, so it will be removed in the Torch IR to ATen IR transformation, thus losing all multi-stream scheduling.⛔ Skipped due to learnings
Learnt from: amitz-nv PR: NVIDIA/TensorRT-LLM#6968 File: cpp/tensorrt_llm/thop/loraOp.cpp:133-141 Timestamp: 2025-08-17T15:07:01.420Z Learning: In TensorRT-LLM's LoRA implementation, the LoraImpl::run() method handles setStream() internally in _runGemm() (line 51 in lora.cpp), along with setWorkspace(). The stream parameter flows from loraOp.cpp through LoraImpl::run() to _runGemm() where setStream() is called appropriately. Adding setStream() in loraOp.cpp would be redundant and goes against the intended architectural design.
docs/source/torch/features/torch_compile_and_piecewise_cuda_graph.md
Outdated
Show resolved
Hide resolved
docs/source/torch/features/torch_compile_and_piecewise_cuda_graph.md
Outdated
Show resolved
Hide resolved
docs/source/torch/features/torch_compile_and_piecewise_cuda_graph.md
Outdated
Show resolved
Hide resolved
docs/source/torch/features/torch_compile_and_piecewise_cuda_graph.md
Outdated
Show resolved
Hide resolved
docs/source/torch/features/torch_compile_and_piecewise_cuda_graph.md
Outdated
Show resolved
Hide resolved
yizhang-nv
changed the title
yizhang-nv
force-pushed
the
doc-for-torch-compile
branch
from
e7cc4df to
0d0170b
Compare
docs/source/torch/features/torch_compile_and_piecewise_cuda_graph.md
Outdated
Show resolved
Hide resolved
docs/source/torch/features/torch_compile_and_piecewise_cuda_graph.md
Outdated
Show resolved
Hide resolved
yizhang-nv
force-pushed
the
doc-for-torch-compile
branch
from
0d0170b to
b44135d
Compare
docs/source/torch/features/torch_compile_and_piecewise_cuda_graph.md
Outdated
Show resolved
Hide resolved
yizhang-nv
force-pushed
the
doc-for-torch-compile
branch
from
4bf2cde to
f14dda8
Compare
yizhang-nv
changed the title
yizhang-nv
force-pushed
the
doc-for-torch-compile
branch
5 times, most recently
from
f8d230d to
6e4c34c
Compare
|
/bot run |
|
PR_Github #22161 [ run ] triggered by Bot. Commit: |
|
PR_Github #22161 [ run ] completed with state |
yizhang-nv
force-pushed
the
doc-for-torch-compile
branch
from
6e4c34c to
053d726
Compare
|
/bot run |
|
PR_Github #22383 [ run ] triggered by Bot. Commit: |
docs/source/torch/features/torch_compile_and_piecewise_cuda_graph.md
Outdated
Show resolved
Hide resolved
docs/source/torch/features/torch_compile_and_piecewise_cuda_graph.md
Outdated
Show resolved
Hide resolved
|
PR_Github #22383 [ run ] completed with state |
yizhang-nv
force-pushed
the
doc-for-torch-compile
branch
2 times, most recently
from
d7ded5a to
307b861
Compare
|
/bot run |
|
PR_Github #22593 [ run ] triggered by Bot. Commit: |
|
PR_Github #22593 [ run ] completed with state |
yizhang-nv
force-pushed
the
doc-for-torch-compile
branch
from
307b861 to
867570a
Compare
|
/bot run --disable-fail-fast |
|
PR_Github #22847 [ run ] triggered by Bot. Commit: |
|
PR_Github #22847 [ run ] completed with state |
yizhang-nv
force-pushed
the
doc-for-torch-compile
branch
2 times, most recently
from
7ece877 to
c36ac12
Compare
yizhang-nv
force-pushed
the
doc-for-torch-compile
branch
from
c36ac12 to
989ddae
Compare
|
/bot skip --comment "Failed test case is not related to this pr" |
|
PR_Github #22977 [ skip ] triggered by Bot. Commit: |
|
PR_Github #22977 [ skip ] completed with state |
…8527) Signed-off-by: yizhang-nv <187001205+yizhang-nv@users.noreply.github.com>
…8527) Signed-off-by: yizhang-nv <187001205+yizhang-nv@users.noreply.github.com>
…8527) Signed-off-by: yizhang-nv <187001205+yizhang-nv@users.noreply.github.com>
…8527) Signed-off-by: yizhang-nv <187001205+yizhang-nv@users.noreply.github.com>
…8527) Signed-off-by: yizhang-nv <187001205+yizhang-nv@users.noreply.github.com> Signed-off-by: FredricZ-2007 <226039983+fredricz-20070104@users.noreply.github.com>
6 participants
Summary by CodeRabbit
Documentation
Chores
Description
Test Coverage
PR Checklist
Please review the following before submitting your PR:
PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.
PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.
Test cases are provided for new code paths (see test instructions)
Any new dependencies have been scanned for license and vulnerabilities
CODEOWNERS updated if ownership changes
Documentation updated as needed
The reviewers assigned automatically/manually are appropriate for the PR.
Please check this after reviewing the above items as appropriate for this PR.
GitHub Bot Help
/bot [-h] ['run', 'kill', 'skip', 'reuse-pipeline'] ...Provide a user friendly way for developers to interact with a Jenkins server.
Run
/bot [-h|--help]to print this help message.See details below for each supported subcommand.
Details
run [--reuse-test (optional)pipeline-id --disable-fail-fast --skip-test --stage-list "A10-PyTorch-1, xxx" --gpu-type "A30, H100_PCIe" --test-backend "pytorch, cpp" --add-multi-gpu-test --only-multi-gpu-test --disable-multi-gpu-test --post-merge --extra-stage "H100_PCIe-TensorRT-Post-Merge-1, xxx" --detailed-log --debug(experimental)]Launch build/test pipelines. All previously running jobs will be killed.
--reuse-test (optional)pipeline-id(OPTIONAL) : Allow the new pipeline to reuse build artifacts and skip successful test stages from a specified pipeline or the last pipeline if no pipeline-id is indicated. If the Git commit ID has changed, this option will be always ignored. The DEFAULT behavior of the bot is to reuse build artifacts and successful test results from the last pipeline.--disable-reuse-test(OPTIONAL) : Explicitly prevent the pipeline from reusing build artifacts and skipping successful test stages from a previous pipeline. Ensure that all builds and tests are run regardless of previous successes.--disable-fail-fast(OPTIONAL) : Disable fail fast on build/tests/infra failures.--skip-test(OPTIONAL) : Skip all test stages, but still run build stages, package stages and sanity check stages. Note: Does NOT update GitHub check status.--stage-list "A10-PyTorch-1, xxx"(OPTIONAL) : Only run the specified test stages. Examples: "A10-PyTorch-1, xxx". Note: Does NOT update GitHub check status.--gpu-type "A30, H100_PCIe"(OPTIONAL) : Only run the test stages on the specified GPU types. Examples: "A30, H100_PCIe". Note: Does NOT update GitHub check status.--test-backend "pytorch, cpp"(OPTIONAL) : Skip test stages which don't match the specified backends. Only support [pytorch, cpp, tensorrt, triton]. Examples: "pytorch, cpp" (does not run test stages with tensorrt or triton backend). Note: Does NOT update GitHub pipeline status.--only-multi-gpu-test(OPTIONAL) : Only run the multi-GPU tests. Note: Does NOT update GitHub check status.--disable-multi-gpu-test(OPTIONAL) : Disable the multi-GPU tests. Note: Does NOT update GitHub check status.--add-multi-gpu-test(OPTIONAL) : Force run the multi-GPU tests in addition to running L0 pre-merge pipeline.--post-merge(OPTIONAL) : Run the L0 post-merge pipeline instead of the ordinary L0 pre-merge pipeline.--extra-stage "H100_PCIe-TensorRT-Post-Merge-1, xxx"(OPTIONAL) : Run the ordinary L0 pre-merge pipeline and specified test stages. Examples: --extra-stage "H100_PCIe-TensorRT-Post-Merge-1, xxx".--detailed-log(OPTIONAL) : Enable flushing out all logs to the Jenkins console. This will significantly increase the log volume and may slow down the job.--debug(OPTIONAL) : Experimental feature. Enable access to the CI container for debugging purpose. Note: Specify exactly one stage in thestage-listparameter to access the appropriate container environment. Note: Does NOT update GitHub check status.For guidance on mapping tests to stage names, see
docs/source/reference/ci-overview.mdand the
scripts/test_to_stage_mapping.pyhelper.kill
killKill all running builds associated with pull request.
skip
skip --comment COMMENTSkip testing for latest commit on pull request.
--comment "Reason for skipping build/test"is required. IMPORTANT NOTE: This is dangerous since lack of user care and validation can cause top of tree to break.reuse-pipeline
reuse-pipelineReuse a previous pipeline to validate current commit. This action will also kill all currently running builds associated with the pull request. IMPORTANT NOTE: This is dangerous since lack of user care and validation can cause top of tree to break.