用合成数据提升视频大模型的时间理解能力
Harnessing Synthetic Preference Data for Enhancing Temporal Understanding of Video-LLMs
- 构建合成时间数据集，引导模型关注视频动态
- 在7个基准上实现性能显著提升
- 适合研究视频理解与时序建模的学者
尽管视频大语言模型在通用视频理解任务（如视频描述）中表现优异，但在需要精细时间理解的任务上仍表现不佳。这源于现有微调数据集缺乏视觉复杂性和时间细节，导致模型过度依赖语言推理而非真正理解视频动态。本文提出TimeWarp方法，系统生成目标化合成时间数据集，用于微调模型以增强其对输入视频的关注。我们构建了一个大规模偏好数据集，捕捉常被忽略的复杂时间动态，使模型响应更依赖视觉与时间信息。实验表明，该方法在现有模型上应用后，显著提升了在7个时间理解基准上的性能，验证了数据集的有效性。代码已开源：https://github.com/sameepv21/timewarp。
原文摘要 · Abstract (English)
While Video Large Language Models (Video-LLMs) have demonstrated remarkable performance across general video understanding benchmarks-particularly in video captioning and descriptive tasks-they consistently underperform on tasks that require fine-grained temporal understanding. This limitation arises due to the lack of visual complexity and temporal nuance in current fine-tuning datasets, leading these models to rely heavily on language-based reasoning rather than truly understanding video dynamics. In this work, we propose TimeWarp, a systematic method to create a targeted synthetic temporal dataset to fine-tune the model's responses to encourage it to focus on the given input video. We introduce a large-scale preference dataset, created using TimeWarp, that captures intricate temporal dynamics often overlooked, grounding the model's responses to visual and temporal information. We demonstrate that when our method is applied to existing models, it significantly improves performance on temporal understanding benchmarks, highlighting the effectiveness of our proposed datasets in advancing temporal understanding in Video-LLMs, resulting in an absolute improvement in performance across seven benchmarks. Code is available at https://github.com/sameepv21/timewarp.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。