Skip to content

[Bug] Minimax h3 audio reference doesn't seem to be working #1882

Description

@stduhpf

Git commit

de298c2

Operating System & Version

Windows

GGML backends

HIP

Command-line arguments used

./build/bin/sd-cli.exe -M vid_gen --diffusion-model C:\sd-models\minimax_h3_ref2va_pruned-Q4_K_M.gguf --vae C:\sd-models\minimax_h3_video_vae_fp16.safetensors --audio-vae C:\sd-models\minimax_h3_audio_vae_fp32.safetensors --llm C:\sd-models\qwen3vl_32b_minimax_h3-Q2_K_M.gguf --prompt-file prompt-file.txt --cfg-scale 1.0 -v -H 128 -W 128 --diffusion-fa --offload-to-cpu --fps 24 --steps 20 --color --preview proj --video-frames $(([int]((3.5*24-5)/17))*17+5) --ref-audio out.wav -r .\assets\qwen\example.png

Steps to reproduce

Try for example to run the command above with these two references:
out.wav
Image

And this as prompt-file.txt:

subject_definitions
<Subject 1> is the character from <Picture 1>.
<Picture 1> is the reference image of the character used as the visual anchor for the target video.
<Audio 1> is the source audio providing the dialogue, timbre, and delivery for the target video.

retention_analysis
retention_analysis
<Subject 1> (appears in [Shot 1]): fully_preserved - The character's appearance, clothing, and identity are kept consistent with the reference.
<Picture 1> ([Shot 1] first frame): fully_preserved - The video begins exactly with the composition and subject pose shown in <Picture 1>.
<Audio 1>: full_copy

summary : [audio reuse + reference generation] The target video features <Subject 1> speaking the dialogue from <Audio 1> while maintaining the visual identity, clothing, and facial features established in <Picture 1>.

detailed_description
[Shot 1] For the target video, at 0.00 seconds into the target video, <Picture 1> (from [Shot 1]) is fully referenced. The scene opens with <Subject 1> in the exact pose and framing of <Picture 1>. The camera holds a static shot with a very slight push in. <Subject 1> (S1) looks directly into the lens and begins to speak, her lips and facial muscles moving in perfect synchronization with the dialogue from <Audio 1>

What you expected to happen

I expect the audio of the video to match the reference.

What actually happened

output--.mp4

Audio is completely different, it seems to ignore the reference altogether

Logs / error messages / stack trace

[INFO ] stable-diffusion.cpp:5973 - MiniMax-H3 Ref2VA: 1 image(s), 0 video(s), 1 audio clip(s)
[DEBUG] vae.hpp:142  - VAE Tile size: 16x16
[DEBUG] ggml_extend.hpp:884  - num tiles : 2, 2 
[DEBUG] ggml_extend.hpp:885  - optimal overlap : 0.750000, 0.750000 (targeting 0.250000)
[DEBUG] ggml_extend.hpp:886  - processing 4 tiles
[DEBUG] model_loader.cpp:1045 - loading 118/562 tensors from C:\sd-models\minimax_h3_video_vae_fp16.safetensors
  |##################################################| 118/118 - 960.80MB/s
[INFO ] model_loader.cpp:1309 - loading tensors completed, taking 0.36s (read: 0.21s, memcpy: 0.00s, convert: 0.00s, copy_to_backend: 0.00s)
[DEBUG] model_manager.cpp:409  - model manager prepared params backend buffer (344.04 MB, 118 tensors, RAM)
[DEBUG] model_manager.cpp:506  - model manager staged compute params (344.04 MB, 118 tensors) to ROCm0, taking 0.08s
[DEBUG] ggml_extend.hpp:2244 - minimax_h3_video_vae compute buffer size: 627.13 MB(VRAM)
  |==================================================| 4/4 - 4.88it/s
[DEBUG] vae.hpp:166  - computing vae encode graph completed, taking 2.21s
[DEBUG] model_loader.cpp:1045 - loading 135/917 tensors from C:\sd-models\minimax_h3_audio_vae_fp32.safetensors
  |##################################################| 135/135 - 910.11MB/s
[INFO ] model_loader.cpp:1309 - loading tensors completed, taking 0.36s (read: 0.18s, memcpy: 0.00s, convert: 0.00s, copy_to_backend: 0.00s)
[DEBUG] model_manager.cpp:409  - model manager prepared params backend buffer (329.46 MB, 135 tensors, RAM)
[DEBUG] model_manager.cpp:506  - model manager staged compute params (329.46 MB, 135 tensors) to ROCm0, taking 0.05s
[DEBUG] ggml_extend.hpp:2244 - minimax_h3_audio_vae compute buffer size: 417.04 MB(VRAM)
[INFO ] minimax_h3_audio_vae.hpp:470  - MiniMax-H3 audio VAE encode completed, taking 0.68s

Additional context / environment details

No notable errors messages. Audio reference does get parsed and encoded, it just seems to be ignored by the DiT.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions