arXiv:2512.05513cs.CV2025-12被引 5

评测视频模型在时空上的推理能力,发现现有模型难以准确定位动作发生位置和时间。

Know-Show: Benchmarking Video-Language Models on Spatio-Temporal Grounded Reasoning

  • 构建五种场景的统一评测框架,融合空间与时间维度的推理任务
  • 在2500个高质量问题上测试,发现主流模型表现远低于人类水平
  • 提出无需训练的插件GRAM,通过注意力选择关键帧和时间戳提升推理精度

大型视频语言模型在多模态理解上取得显著进展,但其推理在时空上仍缺乏充分锚定。本文提出Know-Show,一个用于评估时空锚定推理的新基准,即模型在推理动作语义的同时,能基于视觉和时间证据进行定位。该基准涵盖空间(人物、物体、人-物、手-物)与时间维度的五个互补场景,基于Charades、Action Genome和Ego4D构建,包含2500个由人工编写的问题。实验揭示当前视频语言模型在“展示所知”与“知道所见”方面均存在显著差距。为此,我们提出GRAM,一种无需训练的插件,通过基于注意力的视频标记选择与显式时间戳编码,增强模型的细粒度时空推理能力。在Qwen、VideoR1、Gemini和GPT-4o等开放与闭源模型上的大量实验表明,现有模型普遍无法准确关联动作与其时空上下文。Know-Show为视频-语言理解中的锚定推理提供了统一评估标准,并为可解释、可靠的多模态推理系统提供方向。数据集已公开于https://github.com/LUNAProject22/Know-Show,代码将同步发布。

原文摘要 · Abstract (English)

Large Video-Language Models (Video-LMs) have achieved impressive progress in multimodal understanding, yet their reasoning remains weakly grounded in space and time. We present Know-Show, a new benchmark designed to evaluate spatio-temporal grounded reasoning, the ability of a model to reason about actions and their semantics while simultaneously grounding its inferences in visual and temporal evidence. Know-Show unifies reasoning and localization within a single evaluation framework consisting of five complementary scenarios across spatial (person, object, person-object, and hand-object) and temporal dimensions. Built from Charades, Action Genome, and Ego4D with 2.5K high-quality human-authored questions, the benchmark exposes significant gaps between current Video-LMs and human reasoning. To bridge this gap, we propose GRAM, a training-free plug-in that augments Video-LMs with fine-grained grounded reasoning through attention-based video token selection and explicit timestamp encoding. Extensive experiments across open and closed Video-LMs (e.g., Qwen, VideoR1, Gemini, and GPT-4o) reveal that existing models struggle to "show what they know" and vice versa. Know-Show establishes a unified standard for assessing grounded reasoning in video-language understanding and provides insights toward developing interpretable and reliable multimodal reasoning systems. We have released the dataset at https://github.com/LUNAProject22/Know-Show, and the code will be released in the same repository.

视频理解多模态推理时空定位评测基准

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。