用大模型生成奖励，让小模型高效学会智能体行为。
LLM-Based Offline Learning for Embodied Agents via Consistency-Guided Reward Ensemble
- 大模型生成密集奖励，小模型离线学习。
- 在VirtualHome上性能超越其他离线强化学习方法。
- 参数少、训练快，适合资源有限的场景。
将大语言模型（LLM）用于具身智能体学习已成趋势，但存在实际应用限制。本文不直接用LLM作为智能体，而是将其作为训练工具：通过LLM对训练数据中的单个动作提供密集奖励反馈。为此提出一致性引导奖励集成框架（CoREN），解决LLM生成结果与目标环境域的对齐问题。该框架采用时空一致的自适应奖励集成，从训练数据中提取环境相关的奖励信号，从而实现不同环境下的有效离线学习。在VirtualHome基准测试中，CoREN显著优于其他离线强化学习智能体，并达到80亿参数的先进基于LLM智能体的水平，而其智能体策略网络仅1170万参数，且仅在训练阶段使用大模型。
原文摘要 · Abstract (English)
Employing large language models (LLMs) to enable embodied agents has become popular, yet it presents several limitations in practice. In this work, rather than using LLMs directly as agents, we explore their use as tools for embodied agent learning. Specifically, to train separate agents via offline reinforcement learning (RL), an LLM is used to provide dense reward feedback on individual actions in training datasets. In doing so, we present a consistency-guided reward ensemble framework (CoREN), designed for tackling difficulties in grounding LLM-generated estimates to the target environment domain. The framework employs an adaptive ensemble of spatio-temporally consistent rewards to derive domain-grounded rewards in the training datasets, thus enabling effective offline learning of embodied agents in different environment domains. Experiments with the VirtualHome benchmark demonstrate that CoREN significantly outperforms other offline RL agents, and it also achieves comparable performance to state-of-the-art LLM-based agents with 8B parameters, despite CoREN having only 117M parameters for the agent policy network and using LLMs only for training.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。