让机器人抓取更懂物体特性，能自省纠错，抓得准又稳。
Agentic RAG-VLM: Affordance-Aware Retrieval-Augmented Generation with Self-Reflective Planning for Robotic Grasping
- 用物理属性匹配策略，不再只靠外形相似
- 抓取失败时能分类复盘，分三步重试优化
- 适合复杂场景下需要可靠抓取的机器人应用
在杂乱环境中实现通用机器人抓取对部署于非结构化人机空间的机械臂至关重要。现有基于视觉语言模型（VLM）的方法依赖视觉相似性进行物体匹配，忽视了手柄可抓性、材料脆弱性等物理属性，且缺乏空间推理与失败恢复能力，导致在密集或物理特性多样的物体中表现受限。本文提出 Agentic RAG-VLM 框架，融合检索增强生成（RAG）与视觉语言模型（VLM），引入代理式自反思规划，实现语义理解与物理执行的统一。该框架包含三个紧密耦合模块：(1) 分层属性感知检索（HAA-RAG），编码类型、材质、脆弱性及可抓区域四维属性，依据功能适配度而非外观进行策略检索；(2) 场景图约束推理器，从 VLM 感知构建空间关系图，将邻近、遮挡、支撑约束转化为具体的抓取参数调整；(3) 代理自反思流水线，采用14类失败分类和三级自适应重试机制，实现闭环抓取优化。在涵盖单次抓取、交互与长程任务的12项基准测试中，每配置360次试验，总体成功率达78.3%，相比纯VLM基线提升53.3个百分点，证明属性感知检索、场景图推理与代理式恢复协同作用是鲁棒操作的关键。
原文摘要 · Abstract (English)
Generalizable robotic grasping in cluttered environments is essential for deploying manipulators in unstructured human spaces, yet existing VLM-based methods rely on visual similarity for object matching, neglecting physical affordances such as handle graspability and material fragility, and operate open-loop without spatial reasoning or failure recovery, limiting their effectiveness when objects are densely packed or physically diverse. We present Agentic RAG-VLM, a unified framework that bridges VLM-based semantic understanding and physically grounded grasp execution by integrating retrieval-augmented generation (RAG) with vision-language models (VLMs) and agentic self-reflective planning. Agentic RAG-VLM introduces three tightly coupled components: (1) a Hierarchical Affordance-Aware RAG (HAA-RAG) that encodes four-dimensional affordance descriptors, including type, material, fragility, and graspable region, and retrieves strategies by functional affordance compatibility rather than visual appearance; (2) a Scene Graph Constraint Reasoner that constructs spatial relationship graphs from VLM perception and translates proximity, occlusion, and support constraints into concrete grasp parameter adjustments; and (3) an Agentic Self-Reflective Pipeline with a 14-type failure taxonomy and three-level adaptive retry for closed-loop grasp refinement. Evaluated on a 12-task benchmark spanning single-grasp, interactive, and long-horizon scenarios with 360 trials per configuration, Agentic RAG-VLM achieves 78.3 percent overall success, a 53.3 percentage-point absolute gain over VLM-only baselines, demonstrating that affordance-aware retrieval, scene graph reasoning, and agentic recovery are jointly essential for robust manipulation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。