Skip to content

[Multimodal] Always enable hashing mm data #23308

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
vllm-bot merged 9 commits into from
Aug 21, 2025
Merged

[Multimodal] Always enable hashing mm data #23308

vllm-bot merged 9 commits into from
Aug 21, 2025

Conversation

@ywang96
Copy link
Member

@ywang96 ywang96 commented Aug 21, 2025 •
edited by github-actions bot
Loading

Purpose

Hashing MM data is required for prefix caching & feature caching to work with multimodal models and has always been turned on by default since vLLM V1 is released. Since then the community is generally happy with these two features and there's no real reason why a user would want to turn off both of them.

This PR modifies the internal so that now multimodal data is always hashed and mm_hashes will always be present whenever mm_data is. This also facilitates #22711.

A follow-up PR will be allowing users to pass in mm_hashes themselves from the frontend, as suggested in #22044

Test Plan

Test Result

(Optional) Documentation Update

Essential Elements of an Effective PR Description Checklist
  • The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
  • The test plan, such as providing test command.
  • The test results, such as pasting the results comparison before and after, or e2e results
  • (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.

@ywang96
update
279908b 
Signed-off-by: Roger Wang <hey@rogerw.io>
@github-actions
Copy link

github-actions bot commented Aug 21, 2025

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

@mergify mergify bot added deepseek Related to DeepSeek models llama Related to Llama models multi-modality Related to multi-modality (#4194) v1 labels Aug 21, 2025
@ywang96 ywang96 mentioned this pull request Aug 21, 2025
Merged
4 tasks
@ywang96 ywang96 marked this pull request as ready for review August 21, 2025 04:20
DarkLight1337
DarkLight1337 reviewed Aug 21, 2025
View reviewed changes
@ywang96
comments
44526f5 
Signed-off-by: Roger Wang <hey@rogerw.io>
@ywang96
Copy link
Member Author

ywang96 commented Aug 21, 2025

Seems we need to move embedding inputs to cpu before numpy transferring explicitly (https://buildkite.com/vllm/ci/builds/27839#0198caf0-8e8d-479c-8bb8-91308921a501):

vllm/vllm/multimodal/hasher.py

Lines 45 to 46 in 3128240

if isinstance(obj, torch.Tensor):
return cls.item_to_bytes("tensor", obj.numpy())

Yep - done in 08acd67

ywang96 and others added 4 commits August 21, 2025 08:58
@ywang96
update
79e5641 
Signed-off-by: Roger Wang <hey@rogerw.io>
@ywang96
update
33c4b51 
Signed-off-by: Roger Wang <hey@rogerw.io>
@ywang96
cleanup
b0a1f95 
Signed-off-by: Roger Wang <hey@rogerw.io>
@DarkLight1337
Fix Prithvi
71ab9d5 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
@DarkLight1337
Copy link
Member

DarkLight1337 commented Aug 21, 2025

Failing tests are due to connection error -- force-merging

@vllm-bot vllm-bot merged commit 79f05e4 into vllm-project:main Aug 21, 2025
39 of 41 checks passed
PapaGoose pushed a commit to PapaGoose/vllm that referenced this pull request Aug 21, 2025
@ywang96 @DarkLight1337 @DanilaKirichkoMTSAI
[Multimodal] Always enable hashing mm data (vllm-project#23308)
bdf8b8d 
Signed-off-by: Roger Wang <hey@rogerw.io>
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
Co-authored-by: DarkLight1337 <tlleungac@connect.ust.hk>
Signed-off-by: Danila Kirichko <d.kirichko@mts.ai>
djmmoss pushed a commit to djmmoss/vllm that referenced this pull request Aug 21, 2025
@ywang96 @DarkLight1337 @djmmoss
[Multimodal] Always enable hashing mm data (vllm-project#23308)
15195dc 
Signed-off-by: Roger Wang <hey@rogerw.io>
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
Co-authored-by: DarkLight1337 <tlleungac@connect.ust.hk>
Signed-off-by: Duncan Moss <djm.moss@gmail.com>
@ywang96 ywang96 mentioned this pull request Aug 22, 2025
Merged
4 tasks
Xu-Wenqing pushed a commit to Xu-Wenqing/vllm that referenced this pull request Aug 23, 2025
@ywang96 @DarkLight1337 @Xu-Wenqing
[Multimodal] Always enable hashing mm data (vllm-project#23308)
1399363 
Signed-off-by: Roger Wang <hey@rogerw.io>
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
Co-authored-by: DarkLight1337 <tlleungac@connect.ust.hk>
Signed-off-by: root <xwq391974@alibaba-inc.com>
@ywang96 ywang96 mentioned this pull request Aug 26, 2025
Merged
5 tasks
epwalsh pushed a commit to epwalsh/vllm that referenced this pull request Aug 28, 2025
@ywang96 @DarkLight1337 @epwalsh
[Multimodal] Always enable hashing mm data (vllm-project#23308)
deca930 
Signed-off-by: Roger Wang <hey@rogerw.io>
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
Co-authored-by: DarkLight1337 <tlleungac@connect.ust.hk>
xiao-llm pushed a commit to xiao-llm/vllm that referenced this pull request Aug 28, 2025
@ywang96 @DarkLight1337 @amd-xiaoyu12
[Multimodal] Always enable hashing mm data (vllm-project#23308)
8d9fef5 
Signed-off-by: Roger Wang <hey@rogerw.io>
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
Co-authored-by: DarkLight1337 <tlleungac@connect.ust.hk>
Signed-off-by: Xiao Yu <xiao.yu@amd.com>
zhewenl pushed a commit to zhewenl/vllm that referenced this pull request Aug 28, 2025
@ywang96 @DarkLight1337 @zhewenl
[Multimodal] Always enable hashing mm data (vllm-project#23308)
f9034f3 
Signed-off-by: Roger Wang <hey@rogerw.io>
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
Co-authored-by: DarkLight1337 <tlleungac@connect.ust.hk>
mengxingkongzhouhan pushed a commit to mengxingkongzhouhan/vllm that referenced this pull request Aug 30, 2025
@ywang96 @DarkLight1337 @mengxingkongzhouhan
[Multimodal] Always enable hashing mm data (vllm-project#23308)
a2cc2db 
Signed-off-by: Roger Wang <hey@rogerw.io>
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
Co-authored-by: DarkLight1337 <tlleungac@connect.ust.hk>
zhewenl pushed a commit to zhewenl/vllm that referenced this pull request Sep 3, 2025
@ywang96 @DarkLight1337 @zhewenl
[Multimodal] Always enable hashing mm data (vllm-project#23308)
a30df2e 
Signed-off-by: Roger Wang <hey@rogerw.io>
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
Co-authored-by: DarkLight1337 <tlleungac@connect.ust.hk>
FeiDaLI pushed a commit to FeiDaLI/vllm that referenced this pull request Sep 25, 2025
@ywang96 @DarkLight1337 @FeiDaLI
[Multimodal] Always enable hashing mm data (vllm-project#23308)
9f5c9bb 
Signed-off-by: Roger Wang <hey@rogerw.io>
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
Co-authored-by: DarkLight1337 <tlleungac@connect.ust.hk>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@DarkLight1337 DarkLight1337 DarkLight1337 approved these changes

@Isotr0py Isotr0py Isotr0py approved these changes

Assignees

No one assigned

Labels

deepseek Related to DeepSeek models llama Related to Llama models multi-modality Related to multi-modality (#4194) ready ONLY add when PR is ready to merge/full CI is needed v1

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@ywang96 @DarkLight1337 @Isotr0py @vllm-bot