arXiv:2509.02807cs.CV2025-09被引 1

提出新基准,测试视频大模型能否理解运动描述并精准定位动态物体。

PixFoundation 2.0: Do Video Multi-Modal LLMs Use Motion in Visual Grounding?

  • 设计四类运动导向探测方法,聚焦视频中运动与语言的交互理解。
  • 发现现有数据集易被单帧静态特征误导,模型依赖外观而非运动信息。
  • 提出新基准MoCentric-Bench,适合研究视频中时序运动与语言对齐的模型。

多模态大语言模型(MLLMs)在图像与文本任务中表现出色,但其在视频领域的像素级视觉定位能力研究不足。本文探讨视频MLLM是否利用运动信息进行像素级定位,以及能否根据语言描述中的运动模式分割物体。我们指出当前基准的缺陷:单帧即可捕捉运动指代,无需时序推理。为此,提出四种面向运动的探测技术,用于评估模型区分真实运动与虚假运动、理解运动顺序的能力。由此构建了运动中心基准MoCentric-Bench,确保模型评估聚焦于运动与语言的交互,而非被静态外观线索主导。我们建立强单图基线,性能媲美或超越现有方法。进一步探索简单运动适应策略,在新基准上达到当前最优。该工作挑战未来模型提升密集时空定位与像素级视频理解能力。代码与数据将公开于https://github.com/MSiam/PixFoundation-2.0.git。

原文摘要 · Abstract (English)

Multi-modal large language models (MLLMs) have shown impressive generalization across tasks using images and text modalities. While their extension to video has enabled tasks such as video question answering and video captioning, their pixel-level visual grounding abilities are less studied. In this work, we raise the pertinent question of whether motion is used in pixel-level visual grounding and whether video MLLMs can segment objects based on natural language expressions describing their motion patterns. We identify the shortcomings in the current benchmarks, where we show that a single frame can often suffice for capturing the motion referring expression without any temporal reasoning. To address this, we introduce four motion-centric probing techniques, particularly designed for the visual grounding task, to study video MLLMs' ability to identify true motion from a fake one and their ability to grasp the motion order. Consequently, we provide a motion-centric benchmark, MoCentric-Bench. It ensures that video MLLMs are evaluated towards leveraging the interaction between motion and language rather than being dominated by static appearance cues emphasized in existing visual grounding datasets. We further establish strong single-image baselines that are on par with or outperform prior methods. Finally, we explore simple motion-centric adaptation techniques that provide state-of-the-art performance on our MoCentric-Bench. Our motion-centric benchmark, evaluation and findings challenge future models to improve dense spatiotemporal grounding and pixel-level understanding within videos. Code and datasets will be made publicly available at https://github.com/MSiam/PixFoundation-2.0.git.

视频理解视觉定位运动建模多模态

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。