About encoding temporal information

[lihaoxin05]

Thanks for the good work. In terms of encoding temporal information in video frames, ARC-Hunyuan-Video-7B overlays timestamps onto the frames. What is the motivation for this method? Compared to other methods, such as position encoding (e.g., Qwen-VL and Keye-VL) or prepending timestamp tokens before video frame tokens (e.g., Seed1.5-VL), what are the advantanges of this design? Is there any ablation study to validate its advantages?

[geyuying]

Our choice to overlay timestamps directly onto frames was primarily motivated by two factors:

1. Compared to methods like Seed-1.5-VL that prepend timestamp tokens, our approach does not increase the sequence length. Also, it does not require the capability to process interleaved modalities.

2. Our internal ablation studies on video grounding suggested that explicit temporal encoding (like positional encodings) is not always the most effective way to achieve temporal awareness. Also, it is less flexible for videos with non-uniformly sampled frames. Our method naturally handles such cases by "burning" the absolute time into each frame's visual information.

[lihaoxin05]

Thank you for clarifying.