让机器人像人一样从错误中学习,提升长期任务执行能力。
Learning from Trials and Errors: Reflective Test-Time Planning for Embodied LLMs
- 引入测试时反思机制,分三步优化决策:即时反思、事后训练、回溯修正。
- 在多个真实与仿真环境中实现显著性能提升,零样本泛化至真实机器人。
- 适合研究具身智能、长程任务规划及人类反馈强化学习的开发者。
具身大模型赋予机器人高层任务推理能力,但缺乏对错误的反思能力,导致部署过程变成一系列独立试错,错误重复发生而非积累经验。受人类反思实践启发,我们提出反射式测试时规划,整合三种反思模式:行动中反思(test-time scaling)——在执行前通过内部反思生成并评估多个候选动作;行动后反思(test-time training)——基于外部反馈更新内部反思模型与动作策略;以及回溯性反思,使代理能回顾早期决策并结合后见之明进行模型更新,实现更优的长期信用分配。在自建的长周期家庭任务基准和MuJoCo柜子装配基准上的实验显示,该方法显著优于基线模型,且可在无训练条件下泛化至照片级真实环境(HM3D),并在Franka Panda机械臂上完成真实机器人实验。消融实验表明,三种反思机制相互依赖,回溯反思相比逐步外部反馈在更低计算开销下实现更优信用分配。定性分析进一步展示了反思带来的行为纠正效果。
原文摘要 · Abstract (English)
Embodied LLMs endow robots with high-level task reasoning, but they cannot reflect on what went wrong or why, turning deployment into a sequence of independent trials where mistakes repeat rather than accumulate into experience. Drawing upon human reflective practitioners, we introduce Reflective Test-Time Planning, which integrates two modes of reflection: \textit{reflection-in-action}, where the agent uses test-time scaling to generate and score multiple candidate actions using internal reflections before execution; and \textit{reflection-on-action}, which uses test-time training to update both its internal reflection model and its action policy based on external reflections after execution. We also include retrospective reflection, allowing the agent to re-evaluate earlier decisions and perform model updates with hindsight for proper long-horizon credit assignment. Experiments on our newly-designed Long-Horizon Household benchmark and MuJoCo Cupboard Fitting benchmark show significant gains over baseline models, with zero-shot generalization to photorealistic HM3D environments and real-robot experiments on a Franka Panda arm. Ablations confirm that reflection-in-action and reflection-on-action are mutually dependent, and that retrospective reflection achieves better credit assignment than step-wise external feedback at lower computational overhead. Qualitative analyses further highlight behavioral correction through reflection.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。