arXiv:2506.05336cs.CV2025-06被引 14

让视频理解模型精准定位文本描述的时空位置,提升交互准确性。

VideoMolmo: Spatio-Temporal Grounding Meets Pointing

  • 用语言模型生成坐标,再通过掩码融合确保视频连贯性。
  • 在72000个视频-文本对上训练,点位标注超10万次。
  • 适用于细胞追踪、自动驾驶等五类真实场景,通用性强。

时空定位对生物研究、自动驾驶和人机交互至关重要。现有视频方法虽能跟踪,但缺乏大语言模型的深层推理能力,限制了上下文理解与泛化。我们提出VideoMolmo,一种针对文本描述的细粒度时空指代的大型多模态模型。基于Molmo架构,引入时序注意力模块,使每帧依赖前序帧以保证时间一致性。创新的时序掩码融合流程采用SAM2实现双向点传播,显著提升视频序列间的连贯性。该两步分解策略——先由语言模型生成精确指代坐标,再由序列掩码融合模块生成一致分割——不仅降低语言模型负担,也增强可解释性。因缺乏合适数据集,我们构建了包含7.2万个视频-文本对、10万次物体点标注的综合性数据集。为评估泛化能力,提出VPoS-Bench,一个涵盖细胞追踪、第一人称视觉、自动驾驶、视频-图形用户界面交互和机器人五个现实场景的挑战性分布外基准。还在Refer-VOS和Reasoning VOS任务上进行评估。相比现有模型,VideoMolmo在时空指代准确率和推理能力上均有显著提升。代码与模型已开源。

原文摘要 · Abstract (English)

Spatio-temporal localization is vital for precise interactions across diverse domains, from biological research to autonomous navigation and interactive interfaces. Current video-based approaches, while proficient in tracking, lack the sophisticated reasoning capabilities of large language models, limiting their contextual understanding and generalization. We introduce VideoMolmo, a large multimodal model tailored for fine-grained spatio-temporal pointing conditioned on textual descriptions. Building upon the Molmo architecture, VideoMolmo incorporates a temporal module utilizing an attention mechanism to condition each frame on preceding frames, ensuring temporal consistency. Additionally, our novel temporal mask fusion pipeline employs SAM2 for bidirectional point propagation, significantly enhancing coherence across video sequences. This two-step decomposition, i.e., first using the LLM to generate precise pointing coordinates, then relying on a sequential mask-fusion module to produce coherent segmentation, not only simplifies the task for the language model but also enhances interpretability. Due to the lack of suitable datasets, we curate a comprehensive dataset comprising 72k video-caption pairs annotated with 100k object points. To evaluate the generalization of VideoMolmo, we introduce VPoS-Bench, a challenging out-of-distribution benchmark spanning five real-world scenarios: Cell Tracking, Egocentric Vision, Autonomous Driving, Video-GUI Interaction, and Robotics. We also evaluate our model on Referring Video Object Segmentation (Refer-VOS) and Reasoning VOS tasks. In comparison to existing models, VideoMolmo substantially improves spatio-temporal pointing accuracy and reasoning capability. Our code and models are publicly available at https://github.com/mbzuai-oryx/VideoMolmo.

视频理解时空定位多模态点标注

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。