arXiv:2506.23626cs.AI2025-06被引 1

用语言模型自动调整游戏智能体奖励权重,让其自我修正行为表现。

Self-correcting Reward Shaping via Language Models for Reinforcement Learning Agents in Games

  • 用语言模型根据目标行为和历史表现动态优化奖励权重
  • 一次迭代后成功率从9%提升至74%,最终达80%成功率
  • 无需专家干预,适合快速适配新游戏机制的自动化训练

近年来,强化学习在游戏中的应用快速发展,能够生成多样化的智能体行为以改变玩家体验。然而,将强化学习智能体部署到生产环境面临两大挑战:(1) 设计有效奖励函数通常需要专家知识;(2) 当游戏内容或机制变更时,先前调好的奖励权重可能不再最优。针对后者,我们提出一种基于用户定义的语言行为目标,通过语言模型迭代微调智能体奖励权重的自动化方法。语言模型在每轮迭代中根据目标行为和前序训练的表现统计,提出新的权重调整方案。这一闭环过程使语言模型能自我修正并持续优化输出,逐步实现与目标行为更一致的行为,而无需人工奖励工程。我们在竞速任务中评估该方法,结果表明其在多轮迭代中持续提升性能。仅一次迭代后,成功率从9%升至74%。与人类专家手动调参对比,在最终迭代中,语言模型调优的智能体达到80%成功率,平均完成一圈耗时855时间步,接近专家调参的峰值表现(94%成功率,850时间步)。

原文摘要 · Abstract (English)

Reinforcement Learning (RL) in games has gained significant momentum in recent years, enabling the creation of different agent behaviors that can transform a player's gaming experience. However, deploying RL agents in production environments presents two key challenges: (1) designing an effective reward function typically requires an RL expert, and (2) when a game's content or mechanics are modified, previously tuned reward weights may no longer be optimal. Towards the latter challenge, we propose an automated approach for iteratively fine-tuning an RL agent's reward function weights, based on a user-defined language based behavioral goal. A Language Model (LM) proposes updated weights at each iteration based on this target behavior and a summary of performance statistics from prior training rounds. This closed-loop process allows the LM to self-correct and refine its output over time, producing increasingly aligned behavior without the need for manual reward engineering. We evaluate our approach in a racing task and show that it consistently improves agent performance across iterations. The LM-guided agents show a significant increase in performance from $9\%$ to $74\%$ success rate in just one iteration. We compare our LM-guided tuning against a human expert's manual weight design in the racing task: by the final iteration, the LM-tuned agent achieved an $80\%$ success rate, and completed laps in an average of $855$ time steps, a competitive performance against the expert-tuned agent's peak $94\%$ success, and $850$ time steps.

强化学习语言模型奖励设计游戏AI

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。