Migrate Interns1 inputs to TensorSchema #23510

bbeckca · 2025-08-24T21:44:35Z

Purpose

This PR migrates Interns1 inputs from a TypedDict-based definition to a structured TensorSchema model with runtime shape validation. This brings it in line with recent changes to Phi3VImagePixelInputs, and is part of a broader effort to improve input contract enforcement and debug-ability across multi-modal models.

More details: #14764 (comment)

Image Input Classes:

InternS1ImagePixelInputs
InternS1ImageEmbeddingInputs

Video Input Classes:

InternS1VideoPixelInputs
InternS1VideoEmbeddingInputs

Test Plan

Confirm validation works via standalone tests in tests/standalone_test/test_tensor_schema.py and rely on CI to check integration.

Test Result

(venv) benjibeck@Benjis-MacBook-Pro vllm % python3 -m pytest tests/utils_/test_tensor_schema.py -v --log-cli-level=DEBUG
====================================================================================================== test session starts ======================================================================================================
platform darwin -- Python 3.9.6, pytest-8.4.1, pluggy-1.6.0 -- /Users/benjibeck/Projects/vllm/venv/bin/python3
cachedir: .pytest_cache
rootdir: /Users/benjibeck/Projects/vllm
configfile: pyproject.toml
plugins: anyio-4.9.0
collected 19 items                                                                                                                                                                                                              

tests/utils_/test_tensor_schema.py::test_tensor_schema_valid_tensor PASSED                                                                                                                                                [  5%]
tests/utils_/test_tensor_schema.py::test_tensor_schema_optional_fields PASSED                                                                                                                                             [ 10%]
tests/utils_/test_tensor_schema.py::test_tensor_schema_constant_dim_failure PASSED                                                                                                                                        [ 15%]
tests/utils_/test_tensor_schema.py::test_tensor_schema_invalid_types_in_list PASSED                                                                                                                                       [ 21%]
tests/utils_/test_tensor_schema.py::test_tensor_schema_rank_mismatch PASSED                                                                                                                                               [ 26%]
tests/utils_/test_tensor_schema.py::test_tensor_schema_missing_required_field PASSED                                                                                                                                      [ 31%]
tests/utils_/test_tensor_schema.py::test_tensor_schema_symbolic_dim_mismatch PASSED                                                                                                                                       [ 36%]
tests/utils_/test_tensor_schema.py::test_tensor_schema_list_tensor_valid PASSED                                                                                                                                           [ 42%]
tests/utils_/test_tensor_schema.py::test_tensor_schema_variable_patch_counts_valid PASSED                                                                                                                                 [ 47%]
tests/utils_/test_tensor_schema.py::test_tensor_schema_tuple_tensor_valid PASSED                                                                                                                                          [ 52%]
tests/utils_/test_tensor_schema.py::test_tensor_schema_inconsistent_shapes_in_list PASSED                                                                                                                                 [ 57%]
tests/utils_/test_tensor_schema.py::test_tensor_schema_empty_list PASSED                                                                                                                                                  [ 63%]
tests/utils_/test_tensor_schema.py::test_tensor_schema_validation_disabled_skips_shape_check PASSED                                                                                                                       [ 68%]
tests/utils_/test_tensor_schema.py::test_tensor_schema_with_valid_resolve_binding_dims PASSED                                                                                                                             [ 73%]
tests/utils_/test_tensor_schema.py::test_tensor_schema_with_invalid_resolve_binding_dims PASSED                                                                                                                           [ 78%]
tests/utils_/test_tensor_schema.py::test_tensor_schema_with_list_of_symbolic_dim PASSED                                                                                                                                   [ 84%]
tests/utils_/test_tensor_schema.py::test_tensor_schema_with_list_of_symbolic_dim_mismatch_in_length PASSED                                                                                                                [ 89%]
tests/utils_/test_tensor_schema.py::test_valid_tensor_schema_with_static_last_dim PASSED                                                                                                                                  [ 94%]
tests/utils_/test_tensor_schema.py::test_invalid_tensor_schema_with_static_last_dim PASSED                                                                                                                                [100%]

gemini-code-assist

Code Review

This pull request migrates Interns1 inputs to use TensorSchema for better input validation, which is a great improvement for code quality and debug-ability. However, the refactoring has introduced a few critical issues where the num_patches field was either removed from an input class definition or from its instantiation, which will lead to runtime errors. I've provided comments and suggestions to fix these issues.

vllm/model_executor/models/interns1.py

gemini-code-assist · 2025-08-24T21:47:02Z

vllm/model_executor/models/interns1.py

The num_patches field is required by InternS1VideoPixelInputs but it's missing from the constructor call. This will cause a ValueError at runtime because a required field is missing.

Suggested change

return InternS1VideoPixelInputs(

type="pixel_values_videos",

pixel_values=self._validate_pixel_values(

pixel_values_flat_video),

num_patches=video_num_patches,

pixel_values=pixel_values_flat_video,

resolve_bindings={

"h": h,

"w": w,

},

)

return InternS1VideoPixelInputs(

type="pixel_values_videos",

pixel_values=pixel_values_flat_video,

num_patches=video_num_patches,

resolve_bindings={

"h": h,

"w": w,

},

)

bbeckca · 2025-08-25T15:17:49Z

Observing failing MM test for missing field. Will check schema:

[2025-08-25T04:07:07Z] FAILED models/multimodal/processing/test_tensor_schema.py::test_model_tensor_schema[InternS1ForConditionalGeneration-internlm/Intern-S1] - Exception: Call to collective_rpc method failed: Required field 'num_patches' is missing

Signed-off-by: Benji Beck <benjibeck@meta.com>

gemini-code-assist bot reviewed Aug 24, 2025

View reviewed changes

DarkLight1337 approved these changes Aug 25, 2025

View reviewed changes

DarkLight1337 enabled auto-merge (squash) August 25, 2025 02:43

github-actions bot added the ready ONLY add when PR is ready to merge/full CI is needed label Aug 25, 2025

bbeckca added 2 commits August 31, 2025 18:10

Migrate Interns1 inputs to TensorSchema

5d50a9b

Signed-off-by: Benji Beck <benjibeck@meta.com>

Fix num_patches in InternS1ImagePixelInputs

f0d270b

Signed-off-by: Benji Beck <benjibeck@meta.com>

auto-merge was automatically disabled September 1, 2025 01:21
Head branch was pushed to by a user without write access

bbeckca force-pushed the interns1 branch from 35e1b1c to f0d270b Compare September 1, 2025 01:21

DarkLight1337 enabled auto-merge (squash) September 1, 2025 06:06

Add num_patches to InternS1ImagePixelInputs

c59ada4

Signed-off-by: Benji Beck <benjibeck@meta.com>

auto-merge was automatically disabled September 1, 2025 19:50
Head branch was pushed to by a user without write access

Merge branch 'main' into interns1

7298b50

DarkLight1337 enabled auto-merge (squash) September 2, 2025 04:02

DarkLight1337 merged commit 56d0408 into vllm-project:main Sep 2, 2025
42 checks passed

eicherseiji pushed a commit to eicherseiji/vllm that referenced this pull request Sep 9, 2025

Migrate Interns1 inputs to TensorSchema (vllm-project#23510)

460dcaf

Signed-off-by: Benji Beck <benjibeck@meta.com>

FeiDaLI pushed a commit to FeiDaLI/vllm that referenced this pull request Sep 25, 2025

Migrate Interns1 inputs to TensorSchema (vllm-project#23510)

c44e784

Signed-off-by: Benji Beck <benjibeck@meta.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

Migrate Interns1 inputs to TensorSchema #23510

Migrate Interns1 inputs to TensorSchema #23510

Uh oh!

bbeckca commented Aug 24, 2025 •

edited by github-actions bot

Loading

Uh oh!

gemini-code-assist bot left a comment

Uh oh!

Uh oh!

Uh oh!

gemini-code-assist bot Aug 24, 2025

Uh oh!

bbeckca commented Aug 25, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

-            return InternS1VideoPixelInputs(
-                type="pixel_values_videos",
-                pixel_values=self._validate_pixel_values(
-                    pixel_values_flat_video),
-                num_patches=video_num_patches,
-                pixel_values=pixel_values_flat_video,
-                resolve_bindings={
-                    "h": h,
-                    "w": w,
-                },
-            )
+            return InternS1VideoPixelInputs(
+                type="pixel_values_videos",
+                pixel_values=pixel_values_flat_video,
+                num_patches=video_num_patches,
+                resolve_bindings={
+                    "h": h,
+                    "w": w,
+                },
+            )

Uh oh!

Migrate Interns1 inputs to TensorSchema #23510

Migrate Interns1 inputs to TensorSchema #23510

Uh oh!

Conversation

bbeckca commented Aug 24, 2025 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Purpose

Test Plan

Test Result

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

gemini-code-assist bot Aug 24, 2025

Choose a reason for hiding this comment

Uh oh!

bbeckca commented Aug 25, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

bbeckca commented Aug 24, 2025 •

edited by github-actions bot

Loading