用强化学习训练博弈智能体，逼近雅茨基骰子游戏最优策略。
Yahtzee: Reinforcement Learning Techniques for Stochastic Combinatorial Games
- 构建马尔可夫决策过程模型，采用自对弈与多头网络结构训练。
- 平均得分241.78，接近最优分254.59的95%，上区奖励达成率24.9%。
- 发现长期信用分配难题，智能体过度偏好四条组合而非上区保底。
雅茨基骰子是一款具有随机性、组合结构和延迟奖励的经典掷骰游戏，是中等规模强化学习的有趣基准。尽管单人模式可通过动态规划求得最优策略，但多人模式因计算复杂而不可行，需依赖近似方法。本文将雅茨基建模为马尔可夫决策过程（MDP），使用REINFORCE、A2C和PPO三种策略梯度算法训练自对弈智能体，均采用共享主干的多头网络架构。通过消融实验分析特征编码、动作编码、网络结构、回报估计器及熵正则化的影响。在固定训练预算下，REINFORCE与PPO对超参数敏感且未能达到近优性能，而A2C在多种设置下表现稳定。最终智能体在10万场评估游戏中取得中位数241.78分，距离最优动态规划得分254.59仅差5.0%，上区奖励达成率为24.9%，全屋（Yahtzee）成功率为34.1%。所有模型均难以学习上区奖励策略，过度倾向四条组合，凸显长程信用分配与探索挑战。
原文摘要 · Abstract (English)
Yahtzee is a classic dice game with a stochastic, combinatorial structure and delayed rewards, making it an interesting mid-scale RL benchmark. While an optimal policy for solitaire Yahtzee can be computed using dynamic programming methods, multiplayer is intractable, motivating approximation methods. We formulate Yahtzee as a Markov Decision Process (MDP), and train self-play agents using various policy gradient methods: REINFORCE, Advantage Actor-Critic (A2C), and Proximal Policy Optimization (PPO), all using a multi-headed network with a shared trunk. We ablate feature and action encodings, architecture, return estimators, and entropy regularization to understand their impact on learning. Under a fixed training budget, REINFORCE and PPO prove sensitive to hyperparameters and fail to reach near-optimal performance, whereas A2C trains robustly across a range of settings. Our agent attains a median score of 241.78 points over 100,000 evaluation games, within 5.0\% of the optimal DP score of 254.59, achieving the upper section bonus and Yahtzee at rates of 24.9\% and 34.1\%, respectively. All models struggle to learn the upper bonus strategy, overindexing on four-of-a-kind's, highlighting persistent long-horizon credit-assignment and exploration challenges.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。