Skip to content

encode_video truncates a fractional fps, so the video runs longer than its audio #15104

Open

Description

@devanshc777

Describe the bug

diffusers.utils.export_utils.encode_video opens the video stream with rate=int(fps)
(export_utils.py#L364),
so 29.97 is written as 29 fps and 23.976 as 23. The audio stream keeps its real sample rate, so the video ends up
longer than the audio and drifts further out of sync the longer the clip is: about 0.35 s over 10 s at 29.97.

The LTX-2 pipelines take frame_rate as a float and the docs pass it straight through as encode_video(..., fps=frame_rate).
The LTX-2 HDR exporter already keeps the fraction with Fraction(frame_rate).limit_denominator(1000)
(ltx2/export_utils.py#L74).

Reproduction

I can't run the LTX-2 checkpoint on my machine, so this calls encode_video the way the LTX-2 docs do, with inputs
shaped like the pipeline's output_type="np" video and stereo audio:

import av
import numpy as np
import torch

from diffusers.utils.export_utils import encode_video

for fps in (24.0, 29.97, 23.976):
    num_frames = round(fps * 10)  # 10 seconds
    video = np.random.rand(num_frames, 64, 64, 3).astype(np.float32)
    audio = torch.zeros(2, 48000 * 10)
    encode_video(video, fps=fps, output_path="out.mp4", audio=audio, audio_sample_rate=48000)

    with av.open("out.mp4") as container:
        v, a = container.streams.video[0], container.streams.audio[0]
        print(fps, v.average_rate, f"video {float(v.duration * v.time_base):.3f}s", f"audio {float(a.duration * a.time_base):.3f}s")

Logs

24.0 24/1 video 10.000s audio 10.000s
29.97 29/1 video 10.345s audio 10.000s
23.976 23/1 video 10.435s audio 10.000s

System Info

  • diffusers main (29d199a)
  • torch 2.14.1+cpu, PyAV 19.0.2, Python 3.12.13, Windows 11

Proposed fix

Use the same expression as the LTX-2 HDR exporter, rate=Fraction(fps).limit_denominator(1000) (Fraction is already imported in that file), plus a test in tests/others/test_utils.py that encodes 30 frames at 29.97 and checks average_rate == Fraction(2997, 100). The test fails on main with AVRational(29, 1) and passes with the change. With the change the same script prints (300 frames at 29.97 fps really are 10.010 s):

24.0 24/1 video 10.000s audio 10.000s
29.97 2997/100 video 10.010s audio 10.000s
23.976 2997/125 video 10.010s audio 10.000s

Happy to open a PR if that works for you.

Who can help?

@DN6

Activity

  1. devanshc777 commented on Oct 23, 2026

    @devanshc777
    Author

    @DN6 @sayakpaul I have this fixed on a branch with the regression test from the issue, and held off on opening the PR until someone here had a look, as the contributing guide asks. I see #15105 is up for the same change. I'd be glad to be considered for it, but whichever PR is easier for you to merge works for me, and I'm happy to help review either way.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions