通过动态提示与时间对比学习，提升大模型视频时序推理能力。
Temporal Contrastive Learning for Video Temporal Reasoning in Large Vision-Language Models
- 用动态提示生成器捕捉细粒度时序关系
- 在VidSitu数据集上显著提升时序关联与顺序预测性能
- 适合需要精准视频时序理解的场景，如智能剪辑与内容检索
时序推理是视频-语言理解中的关键挑战，要求模型在时间维度上保持语义一致性。现有大型视觉-语言模型（LVLM）和大型语言模型（LLM）在静态任务中表现优异，但在捕捉视频序列中的动态交互与时序依赖方面仍存在不足。本文提出一种名为基于动态提示的时间语义对齐（TSADP）的新框架，通过动态任务特定提示和时间对比学习增强时序推理能力。该框架包含动态提示生成器（DPG），用于编码细粒度时序关系；以及时间对比损失（TCL），用于对齐跨时间的视觉与文本嵌入。我们在引入丰富时序标注的VidSitu数据集上评估该方法，在视频内实体关联、时序关系理解及时间顺序预测等任务中均取得显著优于现有模型的效果。人工评估进一步验证了TSADP生成连贯且语义准确描述的能力。分析表明，该方法具有强鲁棒性、高效率与实际应用价值，为视频-语言理解领域带来重要进展。
原文摘要 · Abstract (English)
Temporal reasoning is a critical challenge in video-language understanding, as it requires models to align semantic concepts consistently across time. While existing large vision-language models (LVLMs) and large language models (LLMs) excel at static tasks, they struggle to capture dynamic interactions and temporal dependencies in video sequences. In this work, we propose Temporal Semantic Alignment via Dynamic Prompting (TSADP), a novel framework that enhances temporal reasoning capabilities through dynamic task-specific prompts and temporal contrastive learning. TSADP leverages a Dynamic Prompt Generator (DPG) to encode fine-grained temporal relationships and a Temporal Contrastive Loss (TCL) to align visual and textual embeddings across time. We evaluate our method on the VidSitu dataset, augmented with enriched temporal annotations, and demonstrate significant improvements over state-of-the-art models in tasks such as Intra-Video Entity Association, Temporal Relationship Understanding, and Chronology Prediction. Human evaluations further confirm TSADP's ability to generate coherent and semantically accurate descriptions. Our analysis highlights the robustness, efficiency, and practical utility of TSADP, making it a step forward in the field of video-language understanding.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。