用强化学习让AI学会跨物体通用推理动作区域。
Affordance-R1: Reinforcement Learning for Generalizable Affordance Reasoning in Multimodal Large Language Model
- 引入思维链引导的相对策略优化,实现认知式推理。
- 零样本泛化性能超越现有方法,测试时具备自主推理能力。
- 适合研究机器人交互与通用视觉理解的开发者。
动作属性定位旨在预测机器人可执行动作所关联的物体特定区域,在人机交互、人物交互、具身操作和具身感知等领域至关重要。现有模型常因缺乏思维链(CoT)推理能力而忽略不同物体间的共性,限制了其在域外场景的泛化与显式推理能力。为此,我们提出Affordance-R1,首个将认知式思维链引导的组相对策略优化(GRPO)融入强化学习框架的统一动作属性定位方法。设计了包含格式、感知与认知奖励的复杂动作函数,有效引导优化方向;并构建高质量的以动作属性为中心的推理数据集ReasonAff支持训练。仅通过强化学习与GRPO训练,无需显式推理数据,Affordance-R1实现了稳健的零样本泛化,并展现出涌现的测试时推理能力。全面实验表明,该模型优于现有主流方法,具备开放世界泛化能力。据我们所知,Affordance-R1是首个将基于GRPO的强化学习与推理融合于动作属性推理的方法。代码与数据集已开源:https://github.com/hq-King/Affordance-R1。
原文摘要 · Abstract (English)
Affordance grounding focuses on predicting the specific regions of objects that are associated with the actions to be performed by robots. It plays a vital role in the fields of human-robot interaction, human-object interaction, embodied manipulation, and embodied perception. Existing models often neglect the affordance shared among different objects because they lack the Chain-of-Thought(CoT) reasoning abilities, limiting their out-of-domain (OOD) generalization and explicit reasoning capabilities. To address these challenges, we propose Affordance-R1, the first unified affordance grounding framework that integrates cognitive CoT guided Group Relative Policy Optimization (GRPO) within a reinforcement learning paradigm. Specifically, we designed a sophisticated affordance function, which contains format, perception, and cognition rewards to effectively guide optimization directions. Furthermore, we constructed a high-quality affordance-centric reasoning dataset, ReasonAff, to support training. Trained exclusively via reinforcement learning with GRPO and without explicit reasoning data, Affordance-R1 achieves robust zero-shot generalization and exhibits emergent test-time reasoning capabilities. Comprehensive experiments demonstrate that our model outperforms well-established methods and exhibits open-world generalization. To the best of our knowledge, Affordance-R1 is the first to integrate GRPO-based RL with reasoning into affordance reasoning. The code of our method and our dataset is released on https://github.com/hq-King/Affordance-R1.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。