Skip to content

Migrate whisper inputs to TensorSchema #23505

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
DarkLight1337 merged 4 commits into from
Sep 3, 2025
Merged

Migrate whisper inputs to TensorSchema #23505

DarkLight1337 merged 4 commits into from
Sep 3, 2025

Conversation

@bbeckca
Copy link
Contributor

@bbeckca bbeckca commented Aug 24, 2025 •
edited by github-actions bot
Loading

Purpose

This PR migrates WhisperAudioInputs inputs from a TypedDict-based definition to a structured TensorSchema model with runtime shape validation. This brings it in line with recent changes to Phi3VImagePixelInputs, and is part of a broader effort to improve input contract enforcement and debug-ability across multi-modal models.

More details: #14764 (comment)

Test Plan

Confirm validation works via standalone tests in tests/standalone_test/test_tensor_schema.py and rely on CI to check integration.

Test Result

(venv) benjibeck@Benjis-MacBook-Pro vllm %  python3 -m pytest tests/utils_/test_tensor_schema.py -v --log-cli-level=DEBUG
====================================================================================================== test session starts ======================================================================================================
platform darwin -- Python 3.9.6, pytest-8.4.1, pluggy-1.6.0 -- /Users/benjibeck/Projects/vllm/venv/bin/python3
cachedir: .pytest_cache
rootdir: /Users/benjibeck/Projects/vllm
configfile: pyproject.toml
plugins: anyio-4.9.0
collected 19 items                                                                                                                                                                                                              

tests/utils_/test_tensor_schema.py::test_tensor_schema_valid_tensor PASSED                                                                                                                                                [  5%]
tests/utils_/test_tensor_schema.py::test_tensor_schema_optional_fields PASSED                                                                                                                                             [ 10%]
tests/utils_/test_tensor_schema.py::test_tensor_schema_constant_dim_failure PASSED                                                                                                                                        [ 15%]
tests/utils_/test_tensor_schema.py::test_tensor_schema_invalid_types_in_list PASSED                                                                                                                                       [ 21%]
tests/utils_/test_tensor_schema.py::test_tensor_schema_rank_mismatch PASSED                                                                                                                                               [ 26%]
tests/utils_/test_tensor_schema.py::test_tensor_schema_missing_required_field PASSED                                                                                                                                      [ 31%]
tests/utils_/test_tensor_schema.py::test_tensor_schema_symbolic_dim_mismatch PASSED                                                                                                                                       [ 36%]
tests/utils_/test_tensor_schema.py::test_tensor_schema_list_tensor_valid PASSED                                                                                                                                           [ 42%]
tests/utils_/test_tensor_schema.py::test_tensor_schema_variable_patch_counts_valid PASSED                                                                                                                                 [ 47%]
tests/utils_/test_tensor_schema.py::test_tensor_schema_tuple_tensor_valid PASSED                                                                                                                                          [ 52%]
tests/utils_/test_tensor_schema.py::test_tensor_schema_inconsistent_shapes_in_list PASSED                                                                                                                                 [ 57%]
tests/utils_/test_tensor_schema.py::test_tensor_schema_empty_list PASSED                                                                                                                                                  [ 63%]
tests/utils_/test_tensor_schema.py::test_tensor_schema_validation_disabled_skips_shape_check PASSED                                                                                                                       [ 68%]
tests/utils_/test_tensor_schema.py::test_tensor_schema_with_valid_resolve_binding_dims PASSED                                                                                                                             [ 73%]
tests/utils_/test_tensor_schema.py::test_tensor_schema_with_invalid_resolve_binding_dims PASSED                                                                                                                           [ 78%]
tests/utils_/test_tensor_schema.py::test_tensor_schema_with_list_of_symbolic_dim PASSED                                                                                                                                   [ 84%]
tests/utils_/test_tensor_schema.py::test_tensor_schema_with_list_of_symbolic_dim_mismatch_in_length PASSED                                                                                                                [ 89%]
tests/utils_/test_tensor_schema.py::test_valid_tensor_schema_with_static_last_dim PASSED                                                                                                                                  [ 94%]
tests/utils_/test_tensor_schema.py::test_invalid_tensor_schema_with_static_last_dim PASSED                                                                                                                                [100%]

gemini-code-assist[bot]
gemini-code-assist bot reviewed Aug 24, 2025
View reviewed changes
Copy link
Contributor

@gemini-code-assist gemini-code-assist bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request successfully migrates WhisperAudioInputs from a TypedDict to TensorSchema, introducing runtime shape validation. This is a good improvement for input contract enforcement and debuggability, aligning with the project's goals. The changes are correct and well-implemented. I have no further comments.

@DarkLight1337 DarkLight1337 enabled auto-merge (squash) August 25, 2025 02:42
@github-actions github-actions bot added the ready ONLY add when PR is ready to merge/full CI is needed label Aug 25, 2025
@bbeckca
Copy link
Contributor Author

bbeckca commented Aug 25, 2025

Observing failing MM test similar to #23503:

[2025-08-25T04:02:55Z]     raise ValueError(f"{field_name} dim[{i}] expected "
[2025-08-25T04:02:55Z] ValueError: input_features dim[1] expected 128, got 80
[2025-08-25T04:02:55Z] FAILED
[2025-08-25T04:02:56Z] tests/models/multimodal/generation/test_whisper.py::test_models[openai/whisper-large-v3-turbo] Fork a new process to run a test 7898

Based on existing schema, this seems to be an issue with the inputs:
schema

auto-merge was automatically disabled September 1, 2025 00:00

Head branch was pushed to by a user without write access

@bbeckca
Copy link
Contributor Author

bbeckca commented Sep 1, 2025

Seeing test failures where _parse_and_validate_audio_input gets input_features=None. This line should be unreachable. @Isotr0py any idea where this is dropped upstream?

python -m pytest tests/models/multimodal/generation/test_whisper.py -k "openai/whisper-large-v3-turbo" -v

@mergify
Copy link

mergify bot commented Sep 1, 2025

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @bbeckca.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

@mergify mergify bot added the needs-rebase label Sep 1, 2025
@bbeckca
Migrate whisper inputs to TensorSchema
4c131dc 
Signed-off-by: Benji Beck <benjibeck@meta.com>
@bbeckca
Set WhisperAudioInputs.input_features as optional
7a28891 
Signed-off-by: Benji Beck <benjibeck@meta.com>
@mergify mergify bot removed the needs-rebase label Sep 1, 2025
@bbeckca
Pad 80 bin input_features to 128 for WhisperAudioInputs
9c0fbd2 
Signed-off-by: Benji Beck <benjibeck@meta.com>
DarkLight1337
DarkLight1337 reviewed Sep 2, 2025
View reviewed changes
vllm/model_executor/models/whisper.py Outdated
input_features = torch.cat(
[feat.to(self.dtype) for feat in input_features])

if input_features.size(1) == 80:
Copy link
Member

@DarkLight1337 DarkLight1337 Sep 2, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we should avoid changing the dimensions like this since it may affect the output of the model. IMO only squeezing or flattening should be allowed. Better to update the shape definitions instead.

@bbeckca
Update annotation for WhisperAudioInputs
666fd92 
Signed-off-by: Benji Beck <benjibeck@meta.com>
@DarkLight1337 DarkLight1337 enabled auto-merge (squash) September 3, 2025 15:48
@DarkLight1337 DarkLight1337 merged commit 731a694 into vllm-project:main Sep 3, 2025
43 checks passed
eicherseiji pushed a commit to eicherseiji/vllm that referenced this pull request Sep 9, 2025
@bbeckca @eicherseiji
Migrate whisper inputs to TensorSchema (vllm-project#23505)
006507a 
Signed-off-by: Benji Beck <benjibeck@meta.com>
FeiDaLI pushed a commit to FeiDaLI/vllm that referenced this pull request Sep 25, 2025
@bbeckca @FeiDaLI
Migrate whisper inputs to TensorSchema (vllm-project#23505)
a136064 
Signed-off-by: Benji Beck <benjibeck@meta.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@DarkLight1337 DarkLight1337 DarkLight1337 approved these changes

+1 more reviewer

@gemini-code-assist gemini-code-assist[bot] gemini-code-assist[bot] left review comments

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

ready ONLY add when PR is ready to merge/full CI is needed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@bbeckca @DarkLight1337