让大模型看懂视频运动,通过轨迹提示增强推理能力
Motion-as-Prompt: Enhancing Motion Reasoning in Multimodal Large Language Models via Motion-Guided Cross-Frame Visual Prompting

- 用运动轨迹生成跨帧视觉提示,补全关键运动信息
- 在CLEVRER和Something-Something-v2上分别提升4.2%和8.9%准确率
- 无需训练或改模型,适合希望增强视频理解的开发者
以运动为中心的视频推理对机器人操作和自动驾驶等交互应用至关重要。然而,多模态大语言模型(MLLMs)通常采用稀疏均匀采样来控制视觉标记和注意力开销,这可能导致丢弃采样帧间的临界变化,限制对物体运动、碰撞和因果关系的推理。为解决此问题,我们提出运动即提示(MaP),一种基于轨迹引导的跨帧视觉提示框架。MaP恢复密集点轨迹,选取运动信息丰富的帧,并将连续采样帧间的轨迹积累直接标注在视觉输入上,使原本隐藏的位移、方向变化和交互对冻结的MLLM可观察。在CLEVRER和Something-Something-v2上的实验表明,MaP持续提升了平均运动推理准确率,分别带来GPT-5.5 4.2%和8.9%的提升。值得注意的是,这些改进未损害非运动理解能力,凸显了MaP的鲁棒性。结果表明,MaP为增强运动中心视频推理提供了一种无需模型训练或架构修改的简单有效方案。
原文摘要 · Abstract (English)
Motion-centric video reasoning is fundamental to interactive applications such as robotic manipulation and autonomous navigation. However, multimodal large language models (MLLMs) typically process videos through sparse uniform sampling to control visual-token and attention costs. This strategy may discard critical transitions between sampled frames, limiting reasoning about object movement, collisions, and causal interactions. To mitigate this issue, we propose Motion-as-Prompt (MaP), a track-guided cross-frame visual prompting framework. MaP recovers dense point trajectories, selects motion-informative frames, and marks the trajectories accumulated between consecutive sampled frames directly onto the visual inputs, making otherwise hidden displacement, direction changes, and interactions observable to frozen MLLMs. Experiments on CLEVRER and Something-Something-v2 show that MaP consistently improves average motion-reasoning accuracy, yielding gains of 4.2% and 8.9% for GPT-5.5, respectively. Notably, these improvements are obtained without degrading non-motion understanding, highlighting the robustness of MaP. These results demonstrate that MaP provides a simple and effective solution for enhancing motion-centric video reasoning without model training or architectural modification. Project page:https://github.com/SunVictor23/MaP.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。