arXiv:2412.11120cs.LGcs.AI2024-12AAAI被引 39

用大模型生成多维隐式奖励,提升强化学习的奖赏分配精度。

Latent Reward: LLM-Empowered Credit Assignment in Episodic Reinforcement Learning

  • 基于大模型生成可执行语义代码,构建多维度隐式奖励
  • 在多个智能体任务中实现更优的时序奖赏分配与贡献划分
  • 适合需要精准奖赏分解的复杂多智能体强化学习场景

强化学习在真实场景中常面临延迟且稀疏的反馈问题,尤其在仅提供回合制奖励的情况下。现有方法虽在奖赏重分配方面取得进展,但仍存在训练困难、归因模糊等问题,主要源于忽视任务表现评估的多维特性。大型语言模型(LLM)蕴含丰富的决策知识,为奖赏重分配提供了可行路径。然而,直接部署LLM面临语言知识与符号形式要求不匹配、推理中固有的随机性与幻觉等问题。为此,我们提出LaRe,一种基于符号的LLM增强决策框架,以改进信用分配。核心是引入隐式奖励概念,作为多维度性能评估机制,从不同视角提升目标达成的可解释性,并促进更有效的奖赏重分配。我们发现,由LLM生成的语义代码能有效连接语言知识与符号化隐式奖励,因其可执行于符号对象。同时,设计了隐式奖励自验证机制以增强推理稳定性。理论上,隐式奖励中无关奖赏的冗余消除有助于提升强化学习性能。大量实验表明,LaRe(i)在时序信用分配上优于现有最优方法,(ii)在多智能体贡献分配上表现优异,(iii)在某些任务中超越使用真实奖赏训练的策略。

原文摘要 · Abstract (English)

Reinforcement learning (RL) often encounters delayed and sparse feedback in real-world applications, even with only episodic rewards. Previous approaches have made some progress in reward redistribution for credit assignment but still face challenges, including training difficulties due to redundancy and ambiguous attributions stemming from overlooking the multifaceted nature of mission performance evaluation. Hopefully, Large Language Model (LLM) encompasses fruitful decision-making knowledge and provides a plausible tool for reward redistribution. Even so, deploying LLM in this case is non-trivial due to the misalignment between linguistic knowledge and the symbolic form requirement, together with inherent randomness and hallucinations in inference. To tackle these issues, we introduce LaRe, a novel LLM-empowered symbolic-based decision-making framework, to improve credit assignment. Key to LaRe is the concept of the Latent Reward, which works as a multi-dimensional performance evaluation, enabling more interpretable goal attainment from various perspectives and facilitating more effective reward redistribution. We examine that semantically generated code from LLM can bridge linguistic knowledge and symbolic latent rewards, as it is executable for symbolic objects. Meanwhile, we design latent reward self-verification to increase the stability and reliability of LLM inference. Theoretically, reward-irrelevant redundancy elimination in the latent reward benefits RL performance from more accurate reward estimation. Extensive experimental results witness that LaRe (i) achieves superior temporal credit assignment to SOTA methods, (ii) excels in allocating contributions among multiple agents, and (iii) outperforms policies trained with ground truth rewards for certain tasks.

强化学习大模型信用分配多智能体

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。