发现大模型生成机器人指令代码存在不可靠性，提出检测与改进方案。
RoboInspector: Unveiling the Unreliability of Policy Code for LLM-enabled Robotic Manipulation
- 构建RoboInspector管道，从任务复杂度和指令粒度分析代码不可靠性
- 在216种组合中识别出4类导致操作失败的不可靠行为
- 基于失败反馈优化代码生成，可靠性提升最高达35%，支持仿真与真实场景
大型语言模型（LLMs）在推理与代码生成方面表现出色，仅需单条指令即可启动机器人操作，由其生成控制机器人的策略代码。然而，由于现实任务需求多样且用户指令复杂，实现可靠的策略代码生成仍面临挑战。不同用户对同一任务给出不同指令，可能导致代码不可靠。为此，我们设计了RoboInspector，一个从任务复杂度和指令粒度两个角度揭示并刻画策略代码不可靠性的流程。我们在两个主流框架中对216种任务、指令与LLM的组合进行了全面实验，识别出导致操作失败的四类主要不可靠行为，并深入分析其成因，为实际开发提供指导。此外，我们提出一种基于失败代码反馈的优化方法，在仿真与真实环境中使策略代码生成可靠性最高提升35%。
原文摘要 · Abstract (English)
Large language models (LLMs) demonstrate remarkable capabilities in reasoning and code generation, enabling robotic manipulation to be initiated with just a single instruction. The LLM carries out various tasks by generating policy code required to control the robot. Despite advances in LLMs, achieving reliable policy code generation remains a significant challenge due to the diverse requirements of real-world tasks and the inherent complexity of user instructions. In practice, different users may provide distinct instructions to drive the robot for the same task, which may cause the unreliability of policy code generation. To bridge this gap, we design \textbf{RoboInspector}, a pipeline to unveil and characterize the unreliability of the policy code for LLM-enabled robotic manipulation from two perspectives: the complexity of the manipulation task and the granularity of the instruction. We perform comprehensive experiments with 216 distinct combinations of tasks, instructions, and LLMs in two prominent frameworks. The \textbf{RoboInspector} identifies four main unreliable behaviors that lead to manipulation failure. We provide a detailed characterization of these behaviors and their underlying causes, giving insight for practical development to reduce unreliability. Furthermore, we introduce a refinement approach guided by failure policy code feedback that improves the reliability of policy code generation by up to 35\% in LLM-enabled robotic manipulation, evaluated in both simulation and real-world environments.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。