Summary

Adds verified CPU support for google/owlvit-base-patch16 zero-shot object detection in fp32 and fp16, and fixes a class-wide exporter defect that could attach keyword-input ONNX names to the wrong tensor shapes. The contribution is Effort L1 / Outcome L1: shared input-name binding code, focused regressions, and exact nested CPU recipes. Goal L2 passes for both precisions; Goal L3 is CLI-BLOCKED because winml eval does not support zero-shot-object-detection.

Model metadata

What the model does

OWL-ViT performs open-vocabulary object detection: it embeds an image and one or more text queries into a shared representation, then returns a class score and normalized bounding box for each image patch so objects can be localized from natural-language labels not fixed at training time.

• Evidence/confidence: pinned checkpoint revision 4b420debb9c806fc4caf9ecc8efb72208c0db892, OwlViTForObjectDetection in Transformers 4.57.6, and its concrete forward/output contract (verified).

Primary user stories

• A user supplies an image and natural-language object labels to obtain scored bounding boxes locating matching objects without training a closed-set detector for those labels. (verified from the pinned model card and concrete Transformers example.)

Supported tasks

The checkpoint declares the task; Transformers resolves OwlViTForObjectDetection and OwlViTProcessor; Optimum registers vendor ONNX support; winml inspect resolves AutoModelForZeroShotObjectDetection, OwlViTOnnxConfig, and the generic WinML inference class.

Model architecture

OwlViTForObjectDetection wraps a CLIP-like dual encoder with 12-layer text and vision Transformers, reshapes 768 × 768 image patches into a 48 × 48 feature map, and applies independent class and box heads to 2,304 patch queries.

OwlViTForObjectDetection
├── OwlViTModel shared multimodal backbone
│   ├── OwlViTTextTransformer: token/position embeddings + encoder blocks x 12 + projection to 512
│   └── OwlViTVisionTransformer: 16 x 16 patch embedding + encoder blocks x 12 + projection to 512
├── Patch feature fusion: vision tokens × broadcast class token + LayerNorm → 48 x 48 x 768
├── OwlViTClassPredictionHead: projected/normalized patch embeddings × text-query embeddings + learned scale/shift
└── OwlViTBoxPredictionHead: MLP box deltas + patch-grid bias + sigmoid

• Source/confidence: pinned checkpoint config and concrete Transformers source (verified).

Validation and support evidence

Baseline

• Current main: 5deebd422e95f28fe8fd912ee50ce874710187b3; WinML CLI 0.2.0.
• Optimum probe: owlvit has vendor tasks feature-extraction and zero-shot-object-detection; WinML adds no override (VENDOR-ONLY).
• Recipe-free CPU build exited 0 and reported Build complete in 101.6s; the graph had 1,007 nodes at opset 17.
• The baseline was not semantically correct: ONNX pixel_values carried INT32[1,16], while input_ids carried FLOAT[1,3,768,768]. Export success therefore did not establish valid input binding.
• Baseline perf emitted mean 1620.309 ms, p50 1620.510 ms, 0.62 samples/s, and RSS delta 1259.54 MB, then exited with Windows status -1073741819; those numbers are runtime-floor evidence only because the graph names were misbound.
• Baseline eval schema exited 2: Task 'zero-shot-object-detection' is not supported by winml eval.
• Starting auto-config correctly described the intended tensors but listed pixel_values before input_ids; keyword invocation was name-safe while ONNX input_names remained positional.

Goal

• Effort: L1 — generalized shared exporter binding plus regression tests.
• Committed Goal ceiling: L3 — CPU makes L0, L1, and L2 reachable; L3 was attempted.
• Outcome: L1 — shared code, tests, and CPU fp32/fp16 recipes.
• Success required semantic named-input structure for both precisions, CPU runtime/perf and memory, named-input PyTorch parity, and either a real task metric or the exact L3 CLI blocker.

Outcome

• Shipped fp32 recipe, fp16 recipe, shared exporter repair, and focused regressions.
• Highest Goal verdict: L2 PASS for fp32 and fp16. L3 CLI-BLOCKED; existing feature gap: Add zero-shot object detection schema and evaluator to winml eval #1147.
• Coverage: full for the chartered CPU tuples; no deferred EP/device/precision tuple.
• Durable OwlViT finding appended separately in gim-home/ModelKitArtifacts#177 (draft Lane A PR).
• Methodology declaration: No methodology friction observed. The current contracts already require the semantic binding, positional-protocol, no-recipe, precision, and blocker checks used here.

Per-EP/device/precision results — including perf and eval data

Goal ladder

Perf

A deterministic named-input CPU harness exited cleanly. winml perf --memory independently produced closely matching complete JSON and the memory phases below before a host-native post-result exit -1073741819; that abnormal exit is retained rather than hidden.

The CLI's independent figures were fp32 mean/p50 1834.173/1886.136 ms, fp16 mean/p50 2319.768/2190.031 ms, and reported model precisions fp32/fp16 respectively.

Eval

Exact blocker: Error: Task 'zero-shot-object-detection' is not supported by winml eval. The task registry gap is tracked by #1147; no dataset or metric was fabricated.

Delta

• src/winml/modelkit/export/htp/exporter.py: HTPExporter._resolve_keyword_input_names() orders ONNX names by the complete, unambiguous inspect.signature(model.forward) intersection for keyword tracing. It falls back safely for incomplete matches and preserves models with explicit get_export_args() positional protocols.
• tests/unit/export/test_pytorch_export.py: verifies exact ONNX name → dtype/shape bindings for mismatched config order and preservation of explicit positional order. The complete file passes: 28 passed.
• Recipe-free acceptance passes: Build complete in 84.2s, with input_ids INT32[1,16], pixel_values FLOAT[1,3,768,768], and attention_mask INT32[1,16].
• The fp32 recipe is the effective current auto-config under the required cpu/cpu coverage path. The fp16 recipe adds quant.mode: fp16; its artifact has 417 FLOAT16 initializers and 304,811,008 external-data bytes versus fp32 609,623,040.
• Reducibility matches the charter: the fix is class-wide and signature-derived, with no model ID or model_type == "owlvit" branch.
• examples/recipes/README.md remains untouched.

Analyze summary — component level and op level

Static public-rule analysis completed for both artifacts; this is compatibility analysis, not runtime execution.

Component-level summary

The unmapped nodes are optimizer-generated reshape/fusion helpers that lost semantic scope names; they remain explicit rather than being guessed into components.

Op-level summary

Rule-less/all-unknown groups: CPU, CUDA, MIGraphX, DML, and VitisAI.

Reproduce commands

$OUT='temp/owlvit-repro'
winml --version
git rev-parse HEAD
winml build -m google/owlvit-base-patch16 -o $OUT/baseline --ep cpu --device cpu --no-analyze --no-optimize --no-quant --no-compile --rebuild --no-color
winml build -c examples/recipes/google_owlvit-base-patch16/cpu/cpu/zero-shot-object-detection_fp32_config.json -m google/owlvit-base-patch16 -o $OUT/fp32 --ep cpu --device cpu --precision fp32 --rebuild --no-color
winml build -c examples/recipes/google_owlvit-base-patch16/cpu/cpu/zero-shot-object-detection_fp16_config.json -m google/owlvit-base-patch16 -o $OUT/fp16 --ep cpu --device cpu --precision fp16 --rebuild --no-color
winml analyze --model $OUT/fp32/model.onnx --ep all --device all --htp-metadata $OUT/fp32/export_htp_metadata.json --output $OUT/analyze-fp32.json --overwrite --no-color
winml perf -m $OUT/fp32/model.onnx --ep cpu --device cpu --precision fp32 --iterations 5 --warmup 1 --memory --no-color
winml perf -m $OUT/fp16/model.onnx --ep cpu --device cpu --precision fp16 --iterations 5 --warmup 1 --memory --no-color
winml eval --schema --task zero-shot-object-detection --no-color