Skip to content

Migrate MiniCPMOAudioInputs to TensorSchema #21847

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
DarkLight1337 merged 3 commits into from
Aug 22, 2025
Merged

Migrate MiniCPMOAudioInputs to TensorSchema #21847

DarkLight1337 merged 3 commits into from
Aug 22, 2025

Conversation

@bbeckca
Copy link
Contributor

@bbeckca bbeckca commented Jul 29, 2025 •
edited by github-actions bot
Loading

Purpose

This PR migrates MiniCPMOAudioInputs from a TypedDict-based definition to a structured TensorSchema model with runtime shape validation. This brings it in line with recent changes to Phi3VImagePixelInputs, and is part of a broader effort to improve input contract enforcement and debug-ability across multi-modal models.

Test Plan

Confirm validation works via standalone tests in tests/standalone_test/test_tensor_schema.py and rely on CI to check integration.

Test Result

(venv) benjibeck@Benjis-MacBook-Pro vllm % python3 -m pytest tests/utils_/test_tensor_schema.py -v --log-cli-level=DEBUG
========================================================================== test session starts ===========================================================================
platform darwin -- Python 3.9.6, pytest-8.4.1, pluggy-1.6.0 -- /Users/benjibeck/Projects/vllm/venv/bin/python3
cachedir: .pytest_cache
rootdir: /Users/benjibeck/Projects/vllm
configfile: pyproject.toml
plugins: anyio-4.9.0
collected 16 items                                                                                                                                                       

tests/utils_/test_tensor_schema.py::test_tensor_schema_valid_tensor PASSED                                                                                         [  6%]
tests/utils_/test_tensor_schema.py::test_tensor_schema_optional_fields PASSED                                                                                      [ 12%]
tests/utils_/test_tensor_schema.py::test_tensor_schema_constant_dim_failure PASSED                                                                                 [ 18%]
tests/utils_/test_tensor_schema.py::test_tensor_schema_symbolic_dim_mismatch PASSED                                                                                [ 25%]
tests/utils_/test_tensor_schema.py::test_tensor_schema_list_tensor_valid PASSED                                                                                    [ 31%]
tests/utils_/test_tensor_schema.py::test_tensor_schema_variable_patch_counts_valid PASSED                                                                          [ 37%]
tests/utils_/test_tensor_schema.py::test_tensor_schema_tuple_tensor_valid PASSED                                                                                   [ 43%]
tests/utils_/test_tensor_schema.py::test_tensor_schema_inconsistent_shapes_in_list PASSED                                                                          [ 50%]
tests/utils_/test_tensor_schema.py::test_tensor_schema_empty_list PASSED                                                                                           [ 56%]
tests/utils_/test_tensor_schema.py::test_tensor_schema_validation_disabled_skips_shape_check PASSED                                                                [ 62%]
tests/utils_/test_tensor_schema.py::test_tensor_schema_with_valid_resolve_binding_dims PASSED                                                                      [ 68%]
tests/utils_/test_tensor_schema.py::test_tensor_schema_with_invalid_resolve_binding_dims PASSED                                                                    [ 75%]
tests/utils_/test_tensor_schema.py::test_tensor_schema_with_list_of_symbolic_dim PASSED                                                                            [ 81%]
tests/utils_/test_tensor_schema.py::test_tensor_schema_with_list_of_symbolic_dim_mismatch_in_length PASSED                                                         [ 87%]
tests/utils_/test_tensor_schema.py::test_valid_tensor_schema_with_static_last_dim PASSED                                                                           [ 93%]
tests/utils_/test_tensor_schema.py::test_invalid_tensor_schema_with_static_last_dim PASSED                                                                         [100%]

============================================================================ warnings summary ============================================================================
venv/lib/python3.9/site-packages/urllib3/__init__.py:35
  /Users/benjibeck/Projects/vllm/venv/lib/python3.9/site-packages/urllib3/__init__.py:35: NotOpenSSLWarning: urllib3 v2 only supports OpenSSL 1.1.1+, currently the 'ssl' module is compiled with 'LibreSSL 2.8.3'. See: https://github.com/urllib3/urllib3/issues/3020
    warnings.warn(

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
===================================================================== 16 passed, 1 warning in 4.38s ======================================================================

@github-actions
Copy link

github-actions bot commented Jul 29, 2025

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

gemini-code-assist[bot]
gemini-code-assist bot reviewed Jul 29, 2025
View reviewed changes
Copy link
Contributor

@gemini-code-assist gemini-code-assist bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request migrates MiniCPMOAudioInputs to use TensorSchema for improved input validation, which is a great step towards better code quality and robustness.

I've found one potential issue with the implementation for MiniCPMOAudioEmbeddingInputs where the shape validation might be too strict and not allow for variable-length slices as documented. My review includes a suggestion to address this.

@bbeckca bbeckca force-pushed the minicpmo branch 4 times, most recently from c8d0aa9 to 741b6ee Compare August 12, 2025 14:35
@bbeckca bbeckca force-pushed the minicpmo branch 2 times, most recently from b2a387a to a566c1d Compare August 19, 2025 14:36
@bbeckca
Copy link
Contributor Author

bbeckca commented Aug 20, 2025

Hi @DarkLight1337 @Isotr0py, just thought I'd check if there's anything I should be doing to unblock landing #21847, #21945, #22020, #22021?

@DarkLight1337 DarkLight1337 added the ready ONLY add when PR is ready to merge/full CI is needed label Aug 20, 2025
DarkLight1337
DarkLight1337 approved these changes Aug 20, 2025
View reviewed changes
Copy link
Member

@DarkLight1337 DarkLight1337 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry for delay, LGTM if test passes

bbeckca and others added 3 commits August 21, 2025 08:03
@bbeckca
Migrate MiniCPMOAudioInputs to TensorSchema
9ed0ae3 
Signed-off-by: Benji Beck <benjibeck@meta.com>
@bbeckca @gemini-code-assist
Update vllm/model_executor/models/minicpmo.py
d9bd550 
Mark `s` as dynamic dim as length of each slice may vary.

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Signed-off-by: Benji Beck <benjibeck@meta.com>
@bbeckca
Set length of audio_features to dynamic dim
f2baec3 
Signed-off-by: Benji Beck <benjibeck@meta.com>
@DarkLight1337 DarkLight1337 merged commit 9987208 into vllm-project:main Aug 22, 2025
41 checks passed
epwalsh pushed a commit to epwalsh/vllm that referenced this pull request Aug 28, 2025
@bbeckca @gemini-code-assist @epwalsh
Migrate MiniCPMOAudioInputs to TensorSchema (vllm-project#21847)
f4dc467 
Signed-off-by: Benji Beck <benjibeck@meta.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
xiao-llm pushed a commit to xiao-llm/vllm that referenced this pull request Aug 28, 2025
@bbeckca @gemini-code-assist @amd-xiaoyu12
Migrate MiniCPMOAudioInputs to TensorSchema (vllm-project#21847)
2dfa590 
Signed-off-by: Benji Beck <benjibeck@meta.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Signed-off-by: Xiao Yu <xiao.yu@amd.com>
zhewenl pushed a commit to zhewenl/vllm that referenced this pull request Aug 28, 2025
@bbeckca @gemini-code-assist @zhewenl
Migrate MiniCPMOAudioInputs to TensorSchema (vllm-project#21847)
5c83480 
Signed-off-by: Benji Beck <benjibeck@meta.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
mengxingkongzhouhan pushed a commit to mengxingkongzhouhan/vllm that referenced this pull request Aug 30, 2025
@bbeckca @gemini-code-assist @mengxingkongzhouhan
Migrate MiniCPMOAudioInputs to TensorSchema (vllm-project#21847)
591a09b 
Signed-off-by: Benji Beck <benjibeck@meta.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
zhewenl pushed a commit to zhewenl/vllm that referenced this pull request Sep 3, 2025
@bbeckca @gemini-code-assist @zhewenl
Migrate MiniCPMOAudioInputs to TensorSchema (vllm-project#21847)
28ed8f6 
Signed-off-by: Benji Beck <benjibeck@meta.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
FeiDaLI pushed a commit to FeiDaLI/vllm that referenced this pull request Sep 25, 2025
@bbeckca @gemini-code-assist @FeiDaLI
Migrate MiniCPMOAudioInputs to TensorSchema (vllm-project#21847)
e821eca 
Signed-off-by: Benji Beck <benjibeck@meta.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@DarkLight1337 DarkLight1337 DarkLight1337 approved these changes

+1 more reviewer

@gemini-code-assist gemini-code-assist[bot] gemini-code-assist[bot] left review comments

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

ready ONLY add when PR is ready to merge/full CI is needed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@bbeckca @DarkLight1337