用大模型指导机器人探索，提升采样效率和成功率
LLM-Guided Task- and Affordance-Level Exploration in Reinforcement Learning
- 双层引导：任务级与可操作性级联合规划
- 在线修正错误计划，实现无需人工监督的多模态探索
- 在真实机器人上实现零样本仿真到现实迁移
强化学习在机器人操作中具有潜力，但面临样本效率低、状态-动作空间过大导致探索困难的问题。现有方法利用大语言模型（LLM）的常识知识和推理能力来引导探索，但生成的计划可能语义合理却物理不可行，导致行为不可靠。本文提出 LLM-TALE 框架，通过 LLM 的规划直接引导强化学习探索。该框架在任务级和可操作性级均集成规划，使代理能够朝语义有意义的动作前进，提升学习效率。不同于以往假设 LLM 计划最优或奖励准确的方法，LLM-TALE 在线修正计划次优性，并在无监督下探索多种可操作性级计划。我们在标准强化学习基准的抓取-放置任务上评估该方法，结果显示其在样本效率和成功率方面优于强基线。真实机器人实验表明具备良好的零样本仿真到现实迁移能力。代码与补充材料见 llm-tale.github.io。
原文摘要 · Abstract (English)
Reinforcement learning (RL) is a promising approach for robotic manipulation, but it can suffer from low sample efficiency and requires extensive exploration of large state-action spaces. Recent methods leverage the commonsense knowledge and reasoning abilities of large language models (LLMs) to guide exploration toward more meaningful states. However, LLMs can produce plans that are semantically plausible yet physically infeasible, yielding unreliable behavior. We introduce LLM-TALE, a framework that uses LLMs' planning to directly steer RL exploration. LLM-TALE integrates planning at both the task level and the affordance level, improving learning efficiency by directing agents toward semantically meaningful actions. Unlike prior approaches that assume optimal LLM-generated plans or rewards, LLM-TALE corrects suboptimality online and explores multimodal affordance-level plans without human supervision. We evaluate LLM-TALE on pick-and-place tasks in standard RL benchmarks, observing improvements in both sample efficiency and success rates over strong baselines. Real-robot experiments indicate promising zero-shot sim-to-real transfer. Code and supplementary material are available at llm-tale.github.io.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。