用大模型生成可读代码实现可解释的逆强化学习。
GRACE: A Language Model Framework for Explainable Inverse Reinforcement Learning
- 用大模型结合进化搜索，从专家轨迹生成可执行的代码化奖励函数。
- 在MuJoCo、BabyAI等基准上准确率高，多任务场景下仍有效。
- 生成的奖励函数可读可验，适合需要透明决策的场景。
逆强化学习旨在从专家示范中恢复奖励模型，但传统方法生成的是难以解释和调试的黑箱模型。本文提出GRACE（Generating Rewards As CodE），利用大语言模型结合进化搜索，直接从专家轨迹反向生成可解释的代码化奖励函数。该奖励函数为可执行代码，可被人工审查与验证。我们在MuJoCo、BabyAI和AndroidWorld基准上进行实证验证，结果表明GRACE能高效学习高精度奖励，即使在复杂多任务环境中亦表现良好。此外，基于其生成的奖励所训练的策略性能优于多种对比的模仿学习和在线强化学习方法（使用真实奖励）。最后，我们展示了GRACE可在多任务设置中构建复杂的奖励API。
原文摘要 · Abstract (English)
Inverse Reinforcement Learning aims to recover reward models from expert demonstrations, but traditional methods yield black-box models that are difficult to interpret and debug. In this work, we introduce GRACE (Generating Rewards As CodE), a method for using Large Language Models within an evolutionary search to reverse-engineer an interpretable, code-based reward function directly from expert trajectories. The resulting reward function is executable code that can be inspected and verified. We empirically validate GRACE on the MuJoCo, BabyAI and AndroidWorld benchmarks, where it efficiently learns highly accurate rewards, even in complex, multi-task settings. Further, we demonstrate that the resulting reward leads to strong policies, compared to both competitive Imitation Learning and online RL approaches with ground-truth rewards. Finally, we show that GRACE is able to build complex reward APIs in multi-task setups.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。