Skip to content

[CPU] Refactor CPU W8A8 scaled_mm #23071

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
bigPYJ1151 merged 6 commits into from
Aug 21, 2025
Merged

[CPU] Refactor CPU W8A8 scaled_mm #23071

bigPYJ1151 merged 6 commits into from
Aug 21, 2025

Conversation

@bigPYJ1151
Copy link
Member

@bigPYJ1151 bigPYJ1151 commented Aug 18, 2025 •
edited by github-actions bot
Loading

Purpose

Refactor CPU W8A8 scaled_mm to:

  • allow weight prepack
  • reduce kernel launching overhead.

cc @Akashcodes732 @nishith-fujitsu It will be better if you can verify this on POWER and ARM, thanks!

this

============ Serving Benchmark Result ============
Successful requests:                     16        
Maximum request concurrency:             8         
Benchmark duration (s):                  85.97     
Total input tokens:                      16368     
Total generated tokens:                  16384     
Request throughput (req/s):              0.19      
Output token throughput (tok/s):         190.57    
Total Token throughput (tok/s):          380.96    
---------------Time to First Token----------------
Mean TTFT (ms):                          1438.26   
Median TTFT (ms):                        1403.56   
P99 TTFT (ms):                           1995.45   
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          40.59     
Median TPOT (ms):                        40.65     
P99 TPOT (ms):                           41.04     
---------------Inter-token Latency----------------
Mean ITL (ms):                           40.59     
Median ITL (ms):                         39.91     
P99 ITL (ms):                            45.53     
==================================================

main

============ Serving Benchmark Result ============
Successful requests:                     16        
Maximum request concurrency:             8         
Benchmark duration (s):                  169.38    
Total input tokens:                      16368     
Total generated tokens:                  16384     
Request throughput (req/s):              0.09      
Output token throughput (tok/s):         96.73     
Total Token throughput (tok/s):          193.36    
---------------Time to First Token----------------
Mean TTFT (ms):                          1410.84   
Median TTFT (ms):                        1545.54   
P99 TTFT (ms):                           2112.74   
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          81.37     
Median TPOT (ms):                        81.23     
P99 TPOT (ms):                           82.66     
---------------Inter-token Latency----------------
Mean ITL (ms):                           81.37     
Median ITL (ms):                         81.39     
P99 ITL (ms):                            90.13     
==================================================

Test Plan

unit tests

Test Result

Pass

(Optional) Documentation Update

Essential Elements of an Effective PR Description Checklist
  • The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
  • The test plan, such as providing test command.
  • The test results, such as pasting the results comparison before and after, or e2e results
  • (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.

@github-actions
Copy link

github-actions bot commented Aug 18, 2025

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

@mergify mergify bot added the ci/build label Aug 18, 2025
gemini-code-assist[bot]
gemini-code-assist bot reviewed Aug 18, 2025
View reviewed changes
Copy link
Contributor

@gemini-code-assist gemini-code-assist bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request is a significant and well-executed refactoring of the CPU W8A8 scaled_mm implementation. The introduction of oneDNN with pre-packed weights and cached primitives is a great step towards improving performance, as demonstrated by the benchmark results. The new CPUScaledMMLinearKernel and the handler-based approach provide a much cleaner and more efficient architecture. The code is well-structured and the addition of comprehensive tests is commendable. I've identified one critical issue regarding tensor memory layout that could lead to runtime errors, which I've detailed in a specific comment.

@bigPYJ1151 bigPYJ1151 mentioned this pull request Aug 18, 2025
Closed
@bigPYJ1151
refactor dnnl w8a8
e581210 
Signed-off-by: jiang1.li <jiang1.li@intel.com>
@bigPYJ1151
fix
c4dd2e3 
Signed-off-by: jiang1.li <jiang1.li@intel.com>
@bigPYJ1151
fix
36de07d 
Signed-off-by: jiang1.li <jiang1.li@intel.com>
@bigPYJ1151
fix
645a9ab 
Signed-off-by: jiang1.li <jiang1.li@intel.com>
@bigPYJ1151
fix
97fcc0d 
Signed-off-by: jiang1.li <jiang1.li@intel.com>
@bigPYJ1151
fix
ed9d785 
Signed-off-by: jiang1.li <jiang1.li@intel.com>
@jikunshang jikunshang added the ready ONLY add when PR is ready to merge/full CI is needed label Aug 20, 2025
@bigPYJ1151 bigPYJ1151 merged commit 7be5d11 into vllm-project:main Aug 21, 2025
82 checks passed
djmmoss pushed a commit to djmmoss/vllm that referenced this pull request Aug 21, 2025
@bigPYJ1151 @djmmoss
[CPU] Refactor CPU W8A8 scaled_mm (vllm-project#23071)
a0c60ea 
Signed-off-by: jiang1.li <jiang1.li@intel.com>
Signed-off-by: Duncan Moss <djm.moss@gmail.com>
epwalsh pushed a commit to epwalsh/vllm that referenced this pull request Aug 28, 2025
@bigPYJ1151 @epwalsh
[CPU] Refactor CPU W8A8 scaled_mm (vllm-project#23071)
37b26bc 
Signed-off-by: jiang1.li <jiang1.li@intel.com>
xiao-llm pushed a commit to xiao-llm/vllm that referenced this pull request Aug 28, 2025
@bigPYJ1151 @amd-xiaoyu12
[CPU] Refactor CPU W8A8 scaled_mm (vllm-project#23071)
5d9760d 
Signed-off-by: jiang1.li <jiang1.li@intel.com>
Signed-off-by: Xiao Yu <xiao.yu@amd.com>
zhewenl pushed a commit to zhewenl/vllm that referenced this pull request Aug 28, 2025
@bigPYJ1151 @zhewenl
[CPU] Refactor CPU W8A8 scaled_mm (vllm-project#23071)
d527ab9 
Signed-off-by: jiang1.li <jiang1.li@intel.com>
mengxingkongzhouhan pushed a commit to mengxingkongzhouhan/vllm that referenced this pull request Aug 30, 2025
@bigPYJ1151 @mengxingkongzhouhan
[CPU] Refactor CPU W8A8 scaled_mm (vllm-project#23071)
6dcc8ee 
Signed-off-by: jiang1.li <jiang1.li@intel.com>
zhewenl pushed a commit to zhewenl/vllm that referenced this pull request Sep 3, 2025
@bigPYJ1151 @zhewenl
[CPU] Refactor CPU W8A8 scaled_mm (vllm-project#23071)
f73b723 
Signed-off-by: jiang1.li <jiang1.li@intel.com>
FeiDaLI pushed a commit to FeiDaLI/vllm that referenced this pull request Sep 25, 2025
@bigPYJ1151 @FeiDaLI
[CPU] Refactor CPU W8A8 scaled_mm (vllm-project#23071)
9983a05 
Signed-off-by: jiang1.li <jiang1.li@intel.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@jikunshang jikunshang jikunshang approved these changes

@tlrmchlsmth tlrmchlsmth Awaiting requested review from tlrmchlsmth tlrmchlsmth is a code owner

@WoosukKwon WoosukKwon Awaiting requested review from WoosukKwon WoosukKwon is a code owner

@yewentao256 yewentao256 Awaiting requested review from yewentao256 yewentao256 is a code owner

@mgoin mgoin Awaiting requested review from mgoin mgoin is a code owner

@robertgshaw2-redhat robertgshaw2-redhat Awaiting requested review from robertgshaw2-redhat robertgshaw2-redhat is a code owner

+1 more reviewer

@gemini-code-assist gemini-code-assist[bot] gemini-code-assist[bot] left review comments

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

ci/build ready ONLY add when PR is ready to merge/full CI is needed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@bigPYJ1151 @jikunshang