用大模型自动优化奖励函数，让离线模仿学习更高效。
PROF: An LLM-based Reward Code Preference Optimization Framework for Offline Imitation Learning
- 用大模型从自然语言生成可执行的奖励代码
- 无需环境交互，通过偏好排序选出最优奖励函数
- 适合无标注数据的离线强化学习场景
离线模仿学习（offline IL）可在无需显式奖励标注的情况下训练有效策略。现有方法尝试利用少量专家示范对未标注数据集估计奖励，但通常假设轨迹与专家示范的相似性与奖励正相关，过度简化了奖励结构。本文提出PROF框架，利用大语言模型（LLMs）从自然语言描述和单条专家轨迹生成并优化可执行的奖励函数代码。提出奖励偏好排序（RPR）策略，无需环境交互或强化学习训练即可评估和排序奖励函数质量，通过主导得分衡量其与专家偏好的契合度。通过交替执行RPR与基于文本的梯度优化，PROF实现了奖励函数选择与精炼的全流程自动化，用于下游策略学习。在D4RL上的实验表明，PROF在多个数据集与领域上超越或匹配近期强基线，验证了方法的有效性。
原文摘要 · Abstract (English)
Offline imitation learning (offline IL) enables training effective policies without requiring explicit reward annotations. Recent approaches attempt to estimate rewards for unlabeled datasets using a small set of expert demonstrations. However, these methods often assume that the similarity between a trajectory and an expert demonstration is positively correlated with the reward, which oversimplifies the underlying reward structure. We propose PROF, a novel framework that leverages large language models (LLMs) to generate and improve executable reward function codes from natural language descriptions and a single expert trajectory. We propose Reward Preference Ranking (RPR), a novel reward function quality assessment and ranking strategy without requiring environment interactions or RL training. RPR calculates the dominance scores of the reward functions, where higher scores indicate better alignment with expert preferences. By alternating between RPR and text-based gradient optimization, PROF fully automates the selection and refinement of optimal reward functions for downstream policy learning. Empirical results on D4RL demonstrate that PROF surpasses or matches recent strong baselines across numerous datasets and domains, highlighting the effectiveness of our approach.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。