arXiv:2505.11326cs.CVcs.AI2025-05

让视觉语言模型实时生成时间精准的对话,突破传统离线模式限制。

Temporally-Grounded Language Generation: Towards Real-Time Vision-Language Models

  • 通过时序交错架构实现视觉与语言令牌的同步处理
  • 在体育直播和第一视角交互数据集上表现优于基线模型
  • 提出新评估指标TRACE,兼顾语义与时间对齐精度

视觉语言模型(VLMs)在图像描述和视频问答等离线任务中取得显著进展。然而,在实时交互环境中,VLMs需生成不仅语义准确且时间精准的回应。我们识别出两大核心能力:感知更新与情境意识,并提出新基准任务——时序锚定语言生成(TGLG),要求模型根据流式视频生成内容与时机均匹配动态视觉输入的语句。为此,我们从体育转播和第一人称人类交互领域构建评估数据集,并引入新指标TRACE,联合衡量语义相似性与时间对齐度。最后,我们提出视觉语言模型时序交错(VLM-TSI),通过时间同步方式交错视觉与语言标记,实现无需轮次假设的实时语言生成。实验表明VLM-TSI显著优于强基线,但整体性能仍有限,凸显TGLG挑战性并推动实时VLM研究。代码已开源。

原文摘要 · Abstract (English)

Vision-language models (VLMs) have shown remarkable progress in offline tasks such as image captioning and video question answering. However, real-time interactive environments impose new demands on VLMs, requiring them to generate utterances that are not only semantically accurate but also temporally precise. We identify two core capabilities necessary for such settings---\textit{perceptual updating} and \textit{contingency awareness}---and propose a new benchmark task, \textbf{Temporally-Grounded Language Generation (TGLG)}, to evaluate them. TGLG requires models to generate utterances in response to streaming video such that both content and timing align with dynamic visual input. To support this benchmark, we curate evaluation datasets from sports broadcasting and egocentric human interaction domains, and introduce a new metric, \textbf{TRACE}, to evaluate TGLG by jointly measuring semantic similarity and temporal alignment. Finally, we present \textbf{Vision-Language Model with Time-Synchronized Interleaving (VLM-TSI)}, a model that interleaves visual and linguistic tokens in a time-synchronized manner, enabling real-time language generation without relying on turn-based assumptions. Experimental results show that VLM-TSI significantly outperforms a strong baseline, yet overall performance remains modest---highlighting the difficulty of TGLG and motivating further research in real-time VLMs. Our code is available at https://github.com/yukw777/tglg.

视觉语言模型实时生成时序对齐多模态

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。