测试视频大模型在时间理解上的一致性，发现其表现极不稳定。
On the Consistency of Video Large Language Models in Temporal Comprehension
- 通过探测初始定位后响应是否一致，评估模型时间理解可靠性
- 现有模型在不同内容、提问和任务下一致性差，误差率超40%
- 提出显式优化一致性的训练方法，显著提升定位准确率与稳定性
视频大语言模型（Video-LLMs）能够对语言查询进行时间定位并检索视频片段。然而，这种时间理解能力尚未被充分研究或理解。为此，我们开展了一项关于预测一致性的研究——这是衡量时间定位鲁棒性与可信度的关键指标。在模型首次定位视频片段后，我们施加一系列探测，检查其后续回答是否与初始定位保持一致，以此评估其理解的可靠性。结果表明，当前 Video-LLMs 对视频内容、语言查询及任务设置的变化极为敏感，暴露出严重的一致性缺陷。我们进一步探索了常见提示工程与指令微调方法作为解决方案，但发现其改进效果往往不稳定。为此，我们提出事件时间验证微调（event temporal verification tuning），显式建模一致性目标，实验显示该方法在定位准确率与一致性方面均有显著提升。相关数据与代码已开源：https://github.com/minjoong507/Consistency-of-Video-LLM。
原文摘要 · Abstract (English)
Video large language models (Video-LLMs) can temporally ground language queries and retrieve video moments. Yet, such temporal comprehension capabilities are neither well-studied nor understood. So we conduct a study on prediction consistency -- a key indicator for robustness and trustworthiness of temporal grounding. After the model identifies an initial moment within the video content, we apply a series of probes to check if the model's responses align with this initial grounding as an indicator of reliable comprehension. Our results reveal that current Video-LLMs are sensitive to variations in video contents, language queries, and task settings, unveiling severe deficiencies in maintaining consistency. We further explore common prompting and instruction-tuning methods as potential solutions, but find that their improvements are often unstable. To that end, we propose event temporal verification tuning that explicitly accounts for consistency, and demonstrate significant improvements for both grounding and consistency. Our data and code are open-sourced at https://github.com/minjoong507/Consistency-of-Video-LLM.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。