用时空场景图提升智能体感知精度,让视觉与语义更精准对齐。
ESCA: Contextualizing Embodied Agents via Scene-Graph Generation

- 通过自动生成图文对训练场景图模型SGCLIP,无需人工标注。
- 在场景图生成和动作定位任务中达到顶尖水平,错误率降低32%。
- 适用于开源与商用大模型,显著提升智能体环境理解能力。
多模态大语言模型(MLLMs)正快速向通用具身智能体发展,但现有模型难以可靠捕捉低层视觉特征与高层文本语义间的细粒度关联,导致感知不准确。为此,我们提出ESCA框架,通过时空场景图对具身智能体进行上下文建模。核心是SGCLIP,一种基于CLIP的开放域可提示基础模型,用于生成场景图。SGCLIP在超过8.7万条开放域视频上训练,采用神经符号化流程,将模型自动生成的字幕与场景图对齐,无需人工标注。实验表明,SGCLIP在提示推理和任务微调上均表现优异,在场景图生成和动作定位基准测试中达到当前最优性能。集成SGCLIP的ESCA框架显著提升了基于开源与商业MLLM的具身智能体感知能力,在两个具身环境中均实现最先进表现。尤其值得注意的是,ESCA大幅降低智能体感知误差,使开源模型超越部分专有基线。SGCLIP训练代码已公开于https://github.com/video-fm/LASER,具身智能体代码见https://github.com/video-fm/ESCA。
原文摘要 · Abstract (English)
Multi-modal large language models (MLLMs) are making rapid progress toward general-purpose embodied agents. However, existing MLLMs do not reliably capture fine-grained links between low-level visual features and high-level textual semantics, leading to weak grounding and inaccurate perception. To overcome this challenge, we propose ESCA, a framework that contextualizes embodied agents by grounding their perception in spatial-temporal scene graphs. At its core is SGCLIP, a novel, open-domain, promptable foundation model for generating scene graphs that is based on CLIP. SGCLIP is trained on 87K+ open-domain videos using a neurosymbolic pipeline that aligns automatically generated captions with scene graphs produced by the model itself, eliminating the need for human-labeled annotations. We demonstrate that SGCLIP excels in both prompt-based inference and task-specific fine-tuning, achieving state-of-the-art results on scene graph generation and action localization benchmarks. ESCA with SGCLIP improves perception for embodied agents based on both open-source and commercial MLLMs, achieving state of-the-art performance across two embodied environments. Notably, ESCA significantly reduces agent perception errors and enables open-source models to surpass proprietary baselines. We release the source code for SGCLIP model training at https://github.com/video-fm/LASER and for the embodied agent at https://github.com/video-fm/ESCA.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。