arXiv:2410.06166cs.CVcs.CL2024-10ICLR被引 36

用文本数据训练视频模型的时序推理能力,效果超纯视频训练模型。

Temporal Reasoning Transfer from Text to Video

  • 从图文数据合成纯文本时序任务,提升视频模型时序理解能力。
  • 在TempCompass上提升5.3分,超越使用2.8万视频样本训练的模型。
  • 适合做视频理解、时序推理相关研究的开发者参考。

视频大语言模型在视频理解方面表现优异,但在追踪时间变化和推理时间关系方面仍存在困难。以往研究认为这是视觉输入时序编码无效所致,但我们的诊断发现,视频表征已包含足够信息,小探测分类器即可实现完美准确率。令人意外的是,问题根源在于底层大语言模型对时间概念的固有理解困难,这在文本时序问答任务中表现明显。基于此,我们提出文本时序推理迁移方法(T3),从现有图文数据集中合成多样化的纯文本时序推理任务,解决复杂时序场景视频样本稀缺问题。令人惊喜的是,仅使用文本数据训练,T3使LongVA-7B在挑战性TempCompass基准上提升5.3个百分点绝对准确率,超越在28,000个视频样本上训练的ShareGPT4Video-8B。增强后的模型在多个视频基准上表现优异,如在Video-MME的时序推理任务中达到49.7的准确率,优于InternVL-Chat-V1.5-20B和VILA1.5-40B等大规模模型。进一步分析显示,文本与视频时序任务表现高度相关,验证了从文本到视频迁移时序推理的有效性。

原文摘要 · Abstract (English)

Video Large Language Models (Video LLMs) have shown promising capabilities in video comprehension, yet they struggle with tracking temporal changes and reasoning about temporal relationships. While previous research attributed this limitation to the ineffective temporal encoding of visual inputs, our diagnostic study reveals that video representations contain sufficient information for even small probing classifiers to achieve perfect accuracy. Surprisingly, we find that the key bottleneck in Video LLMs' temporal reasoning capability stems from the underlying LLM's inherent difficulty with temporal concepts, as evidenced by poor performance on textual temporal question-answering tasks. Building on this discovery, we introduce the Textual Temporal reasoning Transfer (T3). T3 synthesizes diverse temporal reasoning tasks in pure text format from existing image-text datasets, addressing the scarcity of video samples with complex temporal scenarios. Remarkably, without using any video data, T3 enhances LongVA-7B's temporal understanding, yielding a 5.3 absolute accuracy improvement on the challenging TempCompass benchmark, which enables our model to outperform ShareGPT4Video-8B trained on 28,000 video samples. Additionally, the enhanced LongVA-7B model achieves competitive performance on comprehensive video benchmarks. For example, it achieves a 49.7 accuracy on the Temporal Reasoning task of Video-MME, surpassing powerful large-scale models such as InternVL-Chat-V1.5-20B and VILA1.5-40B. Further analysis reveals a strong correlation between textual and video temporal task performance, validating the efficacy of transferring temporal reasoning abilities from text to video domains.

时序推理视频理解文本迁移LLM

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。