让分割模型像人一样思考:先观察场景,再推理动作意图。
Can Segmentation Models Understand the World? Towards Proactive Affordance Reasoning via Visual Chain-of-Thought

- 模型先主动观察场景,生成语言描述和潜在事件推断
- 在意图级指令下,分割准确率显著优于传统方法
- 适合需要理解动作逻辑的机器人交互与智能系统
当前分割模型通过大语言模型与掩码解码器结合,将复杂语言表达转为掩码,但指令仍局限于目标参考。在真实具身交互中,人类指令常以意图为导向,不指明具体区域。为此,我们提出SegWorld,模型在接收指令前主动进行多层级视觉链式思考:先观察场景,描述可见物体并推断其可能支持的事件;收到指令后,从相关物体出发,经由满足意图的动作,推导到物理交互位置及可执行动作的物体部分。我们将其形式化为概率推理,主动观察提供的语言场景上下文能提升意图级指令下的掩码预测性能。我们构建了一个从高层目标推导物体部件的意图-部件基准测试。实验表明,SegWorld在目标参考指令上达到基线水平,在意图级指令上显著提升。
原文摘要 · Abstract (English)
Recent segmentation models couple large language models (LLMs) with mask decoders to ground complex language expressions into masks, yet their instructions remain target-referential: they describe, constrain, or imply the region to be segmented. However, in real-world embodied interaction, human instructions are often at the intent-level, which includes the desired outcome without naming the region that enables it. To bridge this gap, we introduce SegWorld, where the model reasons about the scene through a multi-level visual chain-of-thought (CoT) before committing to a mask. Before receiving any instructions, it proactively observes the scene, describing visible objects and inferring plausible events they may support. Given an instruction, it continues the chain: from the object relevant to the intent, through the action that satisfies it, to the physical interaction site, the object part that affords the action. We formalize SegWorld as probabilistic inference, in which proactive observation supplies a linguistic scene context that improves mask prediction when instructions are given at the level of intent. We construct an intent-to-part benchmark for evaluating affordance-bearing part segmentation from high-level goals. Experiments show SegWorld matches instruction-driven baselines on target-referential instructions and improves substantially on intent-level ones.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。