Add vLLM image support and prompt/image ablation knobs (port of #347 to main) - #439

Description

Ports #347 (opened against vision) onto main. main already has the vision stack, restructured by the #316–#324 series (evals/vision/, providers/olmo_core_vlm/), so this is a re-implementation on that layout rather than a cherry-pick. Measurements and the review discussion are in #347.

What it adds

1. Images in the inline VLLMProvider. Image requests render through the tokenizer's chat template, which emits the model's own image placeholders, and go to vLLM as multi_modal_data. Lazy image references resolve through common.images.resolve_images, like the other multimodal providers. supports_images = True, so the harness routes vision tasks to it.
   • max_images raises vLLM's per-prompt cap (default 1). An oversized prompt fails with the knob named instead of failing its whole batch inside vLLM.
   • chat_template_kwargs is forwarded to apply_chat_template (e.g. enable_thinking).
   • add_bos_token=False and logprobs() refuse image requests instead of silently dropping the images.
   • Measured on Add vllm image support #347: 2.2× olmo_core on short-answer chart_qa, 4.8× on long-decode charxiv_reasoning. HF-vs-vLLM parity over 8 image-QA tasks: mean |Δ| 0.29, max 0.69, signs mixed.
2. HuggingFaceProvider fixes.
   • multimodal now defaults to auto-detection from the config (auto_map, a vision sub-config, or a known image-text-to-text architecture). A remote-code repo read without trust_remote_code raises an error that names the flag.
   • The text-only generate path now applies the chat template. It read request.prompt, which CHAT requests leave empty, so it fed zero tokens into the model.
3. prompt_style and image_mode on the image-QA tasks (evals/vision/tasks/prompting.py). Both default to the published benchmark, and both were measured on Add vllm image support #347's branch:

   • prompt_style=neutral drops mm_olmo's style tag and asks for a short answer. Without it, non-Molmo models answer in sentences and exact match fails. Qwen3-VL-4B-Instruct on full sets vs the Molmo2 paper's numbers:

     task	molmo style (default)	:neutral	published
     ChartQA	13.88	84.24	84.6
     DocVQA (val)	0.00	94.52	95.3
     VQAv2	6.45	82.25	81.7
     TextVQA	11.15	82.89	81.0
     InfoQA (val)	0.00	78.81	80.3
     RealWorldQA	43.79	72.29	70.9
     AI2D	83.23	83.16	84.1

   • mmmu:cot asks for reasoning with a final Answer: line and scores the last one. MMMU is a reasoning benchmark, and the default prompt forbids reasoning. Qwen3-VL-4B goes 49.56 → 57.89 at 2048 tokens (60.11 at 8192). That is still about 7 points under the published 67.4, so it narrows the gap without reproducing it.

   • image_mode: none drops the image and caption substitutes a description from caption_source, a JSONL of {"example_id", "caption"}. Comparing the two against the real image splits error between perception and knowledge. Molmo v9:

     	real	caption	none
     MMMU	48.44	51.78	40.44
     CharXiv descriptive	63.38	76.00	28.75
     CharXiv reasoning	26.70	31.10	6.20

     MMMU is limited by the language model, CharXiv descriptive by perception, and CharXiv reasoning mostly by reasoning.

   • Variants: :neutral on the short-answer tasks and mmmu; mmmu:cot; :text_only / :oracle_caption on mmmu and both charxiv_* tasks.

   • Only ImageQATask honours these fields. Every other vision family refuses them, so an unmodified benchmark is never reported under an ablation label.

   • The new fields enter TaskConfig.to_dict only when non-default, so existing task hashes are unchanged.

4. Dotted sampling_params overrides merge instead of replacing. On main, -o sampling_params.max_tokens=64 replaces the task's SamplingParams with the bare dict, e.g. gsm8k loses its stop_sequences. Reproduced on main before the fix.
5. OPENAI_API_KEY is stripped. A trailing newline in the key surfaced as a generic "Connection error" from the judges.
6. tools/olmo_core_to_hf/export.py. Converts an OLMo-core DCP checkpoint into a released-Molmo2 HF directory, so vLLM can run our checkpoints. --verify compares the export with the reference repo tensor by tensor.
7. README: provider notes and a "Prompt conventions and image ablations" section.

Deliberately not ported from #347

Type of Change

• Bug fix (non-breaking change that fixes an issue)
• New feature (non-breaking change that adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• Documentation update
• Refactoring (no functional changes)

Testing

• Unit tests pass locally (pytest tests/ --ignore=tests/integration/)

• Integration tests pass (if applicable)

• New tests added for new functionality

• Full unit suite, compared against unmodified main: 2964 passed on this branch vs 2879 on main (before the CharXiv knobs and their 7 tests were dropped). The same 14 tests fail on both because of the local environment (no modal, tree-sitter grammar, one tokenizer test), so there are no regressions.

• New tests (78):

  • vLLM image path: placeholders, multi_modal_data, lazy refs, the max_images and limit_mm_per_prompt interplay, the bos/logprobs guards. These run against the existing fake LLM, with no vLLM install needed.
  • HF auto-detect and the chat template on the text path.
  • The prompt_style/image_mode modes and their failure modes, the refusal in non-image-QA families, and task-hash stability.
  • Dotted sampling_params routing.

• ruff format --check, ruff check and ty check src/ are clean. ty reports the same 29 pre-existing diagnostics as unmodified main, and none new.

• Exporter, run end to end on an OLMo-core stage-2 DCP checkpoint: its output is bitwise-identical to Add vllm image support #347's version of the tool (706/706 tensors, same configs). --verify matches the released Molmo2-4B structure.

• GPU smoke from this branch (olmo-eval beaker launch, 1 GPU each on saturn):

  model / provider	task (seeded limit)	score	notes
  Molmo2-4B / HF (molmo2-4b preset)	chart_qa (100)	0.8100	reference leg
  Molmo2-4B / vLLM	chart_qa (100)	0.8100	98/100 outputs identical to HF; the 2 differences are one digit in a long number. Inference 19.7 s vs 59.8 s.
  Qwen3-VL-4B / vLLM	chart_qa:neutral (100)	0.8000	short answers; Add vllm image support #347 measured 84.24 on the full set
  Qwen3-VL-4B / vLLM	mmmu_pro (60)	0.3267	multi-image path; 60/60 completed

  The Molmo2-on-vLLM leg needs -o provider.trust_remote_code=true -o provider.kwargs.limit_mm_per_prompt.video=0.

• Not re-run here: Add vllm image support #347's full throughput sweep and 8-task parity table.

Checklist

• My code follows the project's style guidelines
• I have performed a self-review of my code
• I have added/updated documentation as needed
• My changes generate no new warnings
• Any dependent changes have been merged and published

🤖 Generated with Claude Code