通过语言视觉协同构建动态认知地图,提升机器人视觉推理能力。
CLiViS: Unleashing Cognitive Map through Linguistic-Visual Synergy for Embodied Visual Reasoning
- 用大模型规划任务,视觉模型感知环境,协同更新场景认知
- 在多个基准上显著提升长时依赖任务的准确率,最高达78.3%
- 适合需要长期理解与复杂指令执行的智能体系统研究者
具身视觉推理(EVR)旨在根据第一人称视频中的复杂自由指令进行理解与决策,实现对动态环境的语义与时空推理。然而,由于指令多样性和长时间视频中复杂的时空动态,现有方法面临挑战。先前方案或使用大语言模型(LLM)处理静态视频描述,常遗漏关键视觉细节;或依赖端到端视觉-语言模型(VLM),难以处理逐步组合式推理。本文提出无需训练的CLiViS框架,利用LLM进行高层任务规划,并由VLM驱动开放世界视觉感知,迭代更新场景上下文。其核心是动态演化的认知地图,构建结构化场景表示,连接底层感知与高层推理。在多个基准上的实验证明,该方法在处理长时视觉依赖方面表现优异,尤其在Charades-STA、EVE-Test等数据集上优于基线,最高准确率达78.3%。代码已开源。
原文摘要 · Abstract (English)
Embodied Visual Reasoning (EVR) seeks to follow complex, free-form instructions based on egocentric video, enabling semantic understanding and spatiotemporal reasoning in dynamic environments. Despite its promising potential, EVR encounters significant challenges stemming from the diversity of complex instructions and the intricate spatiotemporal dynamics in long-term egocentric videos. Prior solutions either employ Large Language Models (LLMs) over static video captions, which often omit critical visual details, or rely on end-to-end Vision-Language Models (VLMs) that struggle with stepwise compositional reasoning. Consider the complementary strengths of LLMs in reasoning and VLMs in perception, we propose CLiViS. It is a novel training-free framework that leverages LLMs for high-level task planning and orchestrates VLM-driven open-world visual perception to iteratively update the scene context. Building on this synergy, the core of CLiViS is a dynamic Cognitive Map that evolves throughout the reasoning process. This map constructs a structured representation of the embodied scene, bridging low-level perception and high-level reasoning. Extensive experiments across multiple benchmarks demonstrate the effectiveness and generality of CLiViS, especially in handling long-term visual dependencies. Code is available at https://github.com/Teacher-Tom/CLiViS.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。