[Kernel] Added flashinfer fp8 per-tensor gemms #22895

nvjullin · 2025-08-14T10:10:24Z

Purpose

Added fp8 gemms from flashinfer.
The added gemms have better or same perf as the original gemms so we use it as the default.
For gemm sizes with small M, the added gemms are marginally faster.
For gemm sizes with large M, the added gemms are much faster.
These are the results for llama3 ISL=OSL=1024 concurrency=128 max_num_batched_tokens=8192 TP1.
As expected, TPOT is roughly the same but TTFT improved by ~13%.

~~Requires flashinfer autotuning, so depends on #22346~~ (merged).
~~Funcionality depends on flashinfer PR flashinfer-ai/flashinfer#1479~~ (merged).
~~Perf numbers depends on flashinfer PR flashinfer-ai/flashinfer#1491~~ (merged).
~~Requires next flashinfer release including aforementeiond PRs and vllm updating flashinfer version.~~ (updated flashinfer)

old

============ Serving Benchmark Result ============
Successful requests:                     384       
Maximum request concurrency:             128       
Benchmark duration (s):                  133.38    
Total input tokens:                      392638    
Total generated tokens:                  393216    
Request throughput (req/s):              2.88      
Output token throughput (tok/s):         2948.16   
Total Token throughput (tok/s):          5892.00   
---------------Time to First Token----------------
Mean TTFT (ms):                          2955.22   
Median TTFT (ms):                        1330.74   
P99 TTFT (ms):                           10734.06  
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          40.50     
Median TPOT (ms):                        42.68     
P99 TPOT (ms):                           45.28     
---------------Inter-token Latency----------------
Mean ITL (ms):                           40.50     
Median ITL (ms):                         32.10     
P99 ITL (ms):                            653.22    
==================================================

new

============ Serving Benchmark Result ============
Successful requests:                     384       
Maximum request concurrency:             128       
Benchmark duration (s):                  128.33    
Total input tokens:                      392638    
Total generated tokens:                  393216    
Request throughput (req/s):              2.99      
Output token throughput (tok/s):         3064.22   
Total Token throughput (tok/s):          6123.93   
---------------Time to First Token----------------
Mean TTFT (ms):                          2611.05   
Median TTFT (ms):                        1182.85   
P99 TTFT (ms):                           9319.07   
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          39.19     
Median TPOT (ms):                        41.07     
P99 TPOT (ms):                           44.23     
---------------Inter-token Latency----------------
Mean ITL (ms):                           39.19     
Median ITL (ms):                         31.65     
P99 ITL (ms):                            560.29    
==================================================

lm_eval shows

|Tasks|Version|     Filter     |n-shot|  Metric   |   |Value|   |Stderr|
|-----|------:|----------------|-----:|-----------|---|----:|---|-----:|
|gsm8k|      3|flexible-extract|     5|exact_match|↑  |0.944|±  |0.0103|
|     |       |strict-match    |     5|exact_match|↑  |0.886|±  |0.0142|

Test Plan

Tests to be added.

Test Result

(Optional) Documentation Update

Essential Elements of an Effective PR Description Checklist

The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
The test plan, such as providing test command.
The test results, such as pasting the results comparison before and after, or e2e results
(Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.

gemini-code-assist

Code Review

This pull request introduces support for FlashInfer's FP8 GEMM kernels, which is expected to improve performance, particularly for large batch sizes. The changes primarily involve refactoring the GEMM dispatch logic to accommodate a new 'flashinfer' backend and adding an optimization to pre-calculate combined scales. While the implementation is largely sound, I've identified a critical issue in the new FlashInfer wrapper where the output tensor is not reshaped, potentially causing shape mismatches for inputs with more than two dimensions.

vllm/model_executor/layers/quantization/utils/w8a8_utils.py

github-actions · 2025-08-14T10:22:05Z

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

Signed-off-by: Julien Lin <jullin@nvidia.com>

mgoin · 2025-08-20T19:54:24Z

@nvjullin The Blackwell Test failures look clearly related https://buildkite.com/vllm/ci/builds/27727/steps/canvas?jid=0198c767-0112-49f4-9f26-c9fef601374c#0198c767-0112-49f4-9f26-c9fef601374c/98-3479

vllm/model_executor/layers/quantization/utils/w8a8_utils.py

Signed-off-by: Julien Lin <jullin@nvidia.com>

nvjullin · 2025-08-22T07:59:27Z

@mgoin the remaining errors are all something about huggingface gateway timeout

ProExpertProg

A few minor notes.

This might be a bit too urgent but in general we should really improve the fp8 scaled_mm dispatching. I started a draft pr #19434 but never got around to it.

tests/compile/test_fusion.py

vllm/model_executor/layers/quantization/ptpc_fp8.py

nvpohanh · 2025-08-25T06:28:15Z

@nvjullin Please address the comments and rebase this PR. Thanks!

Signed-off-by: Julien Lin <jullin@nvidia.com>

nvjullin · 2025-08-25T08:42:01Z

This might be a bit too urgent but in general we should really improve the fp8 scaled_mm dispatching. I started a draft pr #19434 but never got around to it.

@mgoin Is there a specific maintainability problem with the current dispatching code you wish to be addressed?

ProExpertProg · 2025-08-25T13:16:45Z

Is there a specific maintainability problem with the current dispatching code you wish to be addressed?

No I think it's out of the scope of this PR. But if you look at dispatching for int8 or Marlin/Machete, that's closer to something we want to do in general when we want to dispatch between multiple possible implementations.

mergify · 2025-08-26T02:43:43Z

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @nvjullin.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

Signed-off-by: Julien Lin <jullin@nvidia.com>

mgoin

LGTM to get in, thanks. We should follow up with using an Enum instead of raw strings

Signed-off-by: Julien Lin <jullin@nvidia.com> Co-authored-by: Michael Goin <mgoin64@gmail.com> Signed-off-by: tc-mb <caitianchi@modelbest.cn>

Signed-off-by: Julien Lin <jullin@nvidia.com> Co-authored-by: Michael Goin <mgoin64@gmail.com>

Signed-off-by: Julien Lin <jullin@nvidia.com> Co-authored-by: Michael Goin <mgoin64@gmail.com> Signed-off-by: Xiao Yu <xiao.yu@amd.com>

Signed-off-by: Julien Lin <jullin@nvidia.com> Co-authored-by: Michael Goin <mgoin64@gmail.com>

nvjullin requested review from mgoin, robertgshaw2-redhat, tlrmchlsmth and yewentao256 as code owners August 14, 2025 10:10

gemini-code-assist bot reviewed Aug 14, 2025

View reviewed changes

vllm/model_executor/layers/quantization/utils/w8a8_utils.py Outdated Show resolved Hide resolved

nvjullin changed the title ~~Added flashinfer fp8 gemms~~ [Kernel] Added flashinfer fp8 gemms Aug 14, 2025

nvjullin changed the title ~~[Kernel] Added flashinfer fp8 gemms~~ [Kernel] Added flashinfer fp8 per-tensor gemms Aug 14, 2025

added flashinfer fp8 gemms

26ddfd9

Signed-off-by: Julien Lin <jullin@nvidia.com>

nvjullin force-pushed the flashinfer-fp8-gemms branch from 4738c59 to 26ddfd9 Compare August 15, 2025 08:43

nvjullin added 2 commits August 15, 2025 10:14

indexing causes torch miscompile

baa5c0e

Signed-off-by: Julien Lin <jullin@nvidia.com>

added tests

1ce5558

Signed-off-by: Julien Lin <jullin@nvidia.com>

nvjullin requested a review from WoosukKwon as a code owner August 18, 2025 07:24

mergify bot added the ci/build label Aug 18, 2025

nvjullin and others added 3 commits August 18, 2025 07:34

fix bad default

14174f1

Signed-off-by: Julien Lin <jullin@nvidia.com>

add to __all__

f2e0334

Signed-off-by: Julien Lin <jullin@nvidia.com>

Merge branch 'main' into flashinfer-fp8-gemms

b860351

mgoin added the ready ONLY add when PR is ready to merge/full CI is needed label Aug 20, 2025

nvpohanh reviewed Aug 21, 2025

View reviewed changes

vllm/model_executor/layers/quantization/utils/w8a8_utils.py Outdated Show resolved Hide resolved

nvjullin added 3 commits August 21, 2025 04:13

fixed test device and added assertions

220382e

Signed-off-by: Julien Lin <jullin@nvidia.com>

limit flashinfer to sm100

2f99b69

Signed-off-by: Julien Lin <jullin@nvidia.com>

moved kernel_warmup to before cuda graph capture

f72ecb1

Signed-off-by: Julien Lin <jullin@nvidia.com>

nvjullin requested review from alexm-redhat, comaniac, njhill and ywang96 as code owners August 21, 2025 09:09

mergify bot added the v1 label Aug 21, 2025

ProExpertProg reviewed Aug 22, 2025

View reviewed changes

tests/compile/test_fusion.py Outdated Show resolved Hide resolved

vllm/model_executor/layers/quantization/ptpc_fp8.py Outdated Show resolved Hide resolved

restrict test to sm100 and changed cutlass_fp8_supported

9a847f7

Signed-off-by: Julien Lin <jullin@nvidia.com>

nvjullin force-pushed the flashinfer-fp8-gemms branch from 09089d8 to 9a847f7 Compare August 25, 2025 08:05

Merge branch 'main' into flashinfer-fp8-gemms

32b5ef8

Signed-off-by: Julien Lin <jullin@nvidia.com>

mergify bot removed the needs-rebase label Aug 25, 2025

fixed typo

85d58a2

Signed-off-by: Julien Lin <jullin@nvidia.com>

mergify bot added the needs-rebase label Aug 26, 2025

Merge branch 'main' into flashinfer-fp8-gemms

a20f927

Signed-off-by: Julien Lin <jullin@nvidia.com>

mergify bot removed the needs-rebase label Aug 26, 2025

mgoin approved these changes Aug 26, 2025

View reviewed changes

simon-mo merged commit f66673a into vllm-project:main Aug 26, 2025
68 of 71 checks passed

nvjullin mentioned this pull request Aug 27, 2025

[Misc] Removed force_fp8_e4m3fnuz from FP8LinearOp #23725

Merged

5 tasks

epwalsh pushed a commit to epwalsh/vllm that referenced this pull request Aug 28, 2025

[Kernel] Added flashinfer fp8 per-tensor gemms (vllm-project#22895)

e6fdca4

Signed-off-by: Julien Lin <jullin@nvidia.com> Co-authored-by: Michael Goin <mgoin64@gmail.com>

zhewenl pushed a commit to zhewenl/vllm that referenced this pull request Aug 28, 2025

[Kernel] Added flashinfer fp8 per-tensor gemms (vllm-project#22895)

92af8eb

Signed-off-by: Julien Lin <jullin@nvidia.com> Co-authored-by: Michael Goin <mgoin64@gmail.com>

zhewenl pushed a commit to zhewenl/vllm that referenced this pull request Sep 3, 2025

[Kernel] Added flashinfer fp8 per-tensor gemms (vllm-project#22895)

38073a9

Signed-off-by: Julien Lin <jullin@nvidia.com> Co-authored-by: Michael Goin <mgoin64@gmail.com>

FeiDaLI pushed a commit to FeiDaLI/vllm that referenced this pull request Sep 25, 2025

[Kernel] Added flashinfer fp8 per-tensor gemms (vllm-project#22895)

c5ce4e7

Signed-off-by: Julien Lin <jullin@nvidia.com> Co-authored-by: Michael Goin <mgoin64@gmail.com>

jasonlizhengjian mentioned this pull request Nov 3, 2025

[Bug]: Async TP pattern matching fails for fp8 models on Blackwell with the default Flashinfer fp8 gemm #27985

Closed

1 task

Uh oh!

[Kernel] Added flashinfer fp8 per-tensor gemms #22895

[Kernel] Added flashinfer fp8 per-tensor gemms #22895

Uh oh!

Conversation

nvjullin commented Aug 14, 2025 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Purpose

old

new

Test Plan

Test Result

(Optional) Documentation Update

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

github-actions bot commented Aug 14, 2025

Uh oh!

mgoin commented Aug 20, 2025

Uh oh!

Uh oh!

nvjullin commented Aug 22, 2025

Uh oh!

ProExpertProg left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

nvpohanh commented Aug 25, 2025

Uh oh!

nvjullin commented Aug 25, 2025

Uh oh!

ProExpertProg commented Aug 25, 2025

Uh oh!

mergify bot commented Aug 26, 2025

Uh oh!

mgoin left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

nvjullin commented Aug 14, 2025 •

edited by github-actions bot

Loading