A bug to fix in the video_inference.py

[dingyue772]

In video_inference.py, there is an inconsistency between video and audio processing when handling videos longer than 150 seconds. Specifically, the logic for computing duration for video and audio is inconsistent.

With a video of length t (where t is a fractional value slightly above 150), the following issue occurs:

• The video sampling logic starts with 150 frames.

• The audio duration is computed as 151.

• Then, at line 129, int() rounds down num_frames to 149.

• This mismatch ultimately causes a size error when padding the audio_embeds tensor.

My current workaround is to modify line 47 from:
duration = vlen / input_fps
to
duration = math.ceil(vlen / input_fps)

Maybe there's a better way to fix the bug? Looking forward to your reply.

[geyuying]

Thank you for raising this issue and for the detailed analysis.

You've correctly identified the root cause: an inconsistency in duration calculation between the video and audio streams, which leads to a synchronization mismatch and subsequent errors.

A more robust solution is to ensure the audio and video are synchronized from the start. We recommend extracting the audio directly from the video file itself.

You can try replacing:

audio, _ = librosa.load(audio_path, sr=sr)

with:

audio, _ = librosa.load(video_path, sr=sr)

Hope this helps.