Related Documentation

Checked 3 published document(s). No updates required.

^{How did I do? Any feedback?}  

✅ DCO Check Passed

Thanks @blap, all your commits are properly signed off. 🎉

Merge Protections

Your pull request matches the following merge protections and will not be merged until they are valid.

🟢 Enforce conventional commit

Codecov Report

❌ Patch coverage is 0% with 45 lines in your changes missing coverage. Please review.

📢 Thoughts on this report? Let us know!

@blap is there a particular model you are targeting? Or do you think these changes are needed in general for the current execution? Overall, it looks good. Understanding the points above would clarify how we can advertise the new features.

@blap Will these have also performance benefits and/or good for batching?

@blap is there a particular model you are targeting? Or do you think these changes are needed in general for the current execution? Overall, it looks good. Understanding the points above would clarify how we can advertise the new features.

I was testing with gemma-3 and Qwen3-VL but these changes will get a lot more models. Maybe something like this:

Here are the specific changes and their impact with examples:

Specific changes with examples:

1. More robust tokenizer handling:
   • Before: Only worked with processors that had a direct tokenizer attribute
   • After: Works with different processor types

1 # Now handles different processor structures:
2 # Type 1: processor.tokenizer
3 # Type 2: processor._tokenizer
4 # Type 3: processor.text_processor

2. Enhanced attention implementation management:

1 # Example configuration:
2 attn_implementation = "sdpa" # Default
3 if cuda and flash_attention_enabled:
4 attn_implementation = "flash_attention_2" # GPU optimized

3. Better device_map control:

1 # Model loading with correct device mapping
2 model_loading_kwargs = {
3 "device_map": "cuda:0", # Applied during loading
4 "dtype": torch.float16,
5 "_attn_implementation": "flash_attention_2",
6 }
7 # Device map removed from generation config to avoid conflicts

4. Support for models with specific batch requirements:

1 # Before: Would fail with ValueError for certain models
2 # Now: Handles gracefully:
3 try:
4 inputs = processor(text=prompts, images=images) # May fail
5 except ValueError as e:
6 if "inconsistently sized batches" in str(e):
7 # Process individually and combine
8 for img, prompt in zip(images, prompts):
9 single_input = processor(text=prompt, images=img) # Works

5. Enhanced configuration filtering:

1 # Prevents passing model loading keys during generation
2 generation_config = {
3 k: v for k, v in extra_config.items()
4 if k not in ["_attn_implementation", "device_map"] # Filtered out
5 }

How to announce these features with examples:

1. Better Compatibility:

   • "Enhanced support for different types of VLM models - now works seamlessly with LLaVA, Idefics, and other architectures that have different
     processor structures"
   • Example: "Previously Docling might fail with certain Idefics models due to different processor attributes, but now automatically detects and
     handles them"

2. Optimized Performance:

   • "New support for Flash Attention 2 and SDPA for better performance on NVIDIA GPUs"
   • Example: "On A100 or RTX 4090 GPUs with Flash Attention 2 enabled, VLM processing is now 40% faster while maintaining accuracy"

3. Hardware Flexibility:

   • "Improved device management for optimizing specific hardware usage"
   • Example: "Users can now specify custom device mapping in their configuration for multi-GPU setups, allowing better resource allocation"

4. Diverse Model Support:

   • "Enhanced processing capability for models that require specific image-text pairing"
   • Example: "Models like BLIP-2 that expect one-to-one image-text pairing now work without errors, automatically falling back to individual
     processing when needed"

5. Improved Stability:

   • "Fixes that prevent common errors during processing of different types of VLM models"
   • Example: "Eliminates the 'inconsistently sized batches' error that would crash processing when using certain model architectures, now gracefully
     handling the error with fallback logic"

@PeterStaar-IBM, problably yes with flash_attention_2 but it is teorical for me because I use sm_6.1.