用固定专家评估轻量级游戏智能体,找出提升实力的关键方法。
A Gold-Standard Study of What Makes a Lightweight Game-Playing Agent Strong
- 用规则专家当衡量标准,不用于训练
- 组合多种策略后胜率从30%升至36%
- 适合想快速构建强博弈智能体的研究者
强化学习智能体在不完美信息纸牌游戏中,其强度取决于训练对手。由于它们击败随机对手超99%时间,且仅与自身副本打平,难以客观评估。因此,我们为金姆拉米(Gin Rummy)构建了一个强大、固定、基于规则的专家,并仅将其作为评估基准,不用于训练。该专家在超过一百次实验中击败所有训练出的智能体,胜率70%至99%。通过系统分析,发现信任域更新、精准奖励设计、渐进式更强对手课程、热启动和保留最佳检查点均有效;叠加使用使自对弈冠军对抗专家的胜率从约30%提升至36%。而短期与长期奖励塑形、学习状态嵌入、模仿学习与DAgger,以及实时大语言模型对手均无效或过重,无法规模化训练。对比MLP、卷积、集合、注意力与循环编码器,表明额外容量无法突破性能上限,说明瓶颈在于信息而非模型规模。我们加入神经虚构自我博弈与信息集蒙特卡洛搜索等基线,确认方法可推广至利德克德州扑克(Leduc Hold'em),其最优解可计算。最终成果是轻量、通用、无需在专家上训练的训练配方,适用于小模型可处理的任何游戏,提供稳健统计并开源可用。
原文摘要 · Abstract (English)
Reinforcement learning agents for imperfect-information card games are only as strong as the opponents they train against, and they are hard to grade, since they beat a random opponent over 99 percent of the time and only tie copies of themselves. So we build a strong, fixed, rule-based expert for Gin Rummy and use it only as a yardstick, never for training. It beats every agent we trained 70 to 99 percent of the time. Across more than a hundred runs, we isolate what makes a lightweight agent stronger. Trust region updates, a well-aimed reward, a curriculum of tougher opponents, warm starting, and keeping the best checkpoint all help, and stacking them lifts a self-play champion from about 30 to 36 percent against the expert. Several ideas did not pay off. Short-term and longer-term reward shaping, learned state embeddings, imitation and DAgger, and a live large language model opponent were each unhelpful, too slow, or too heavy to train at scale. Comparing MLP, convolutional, set-based, attention, and recurrent encoders shows that extra capacity does little to break the ceiling, suggesting the limit is information rather than network size. We add standard baselines (neural fictitious self-play and information set Monte Carlo search) and confirm the approach carries over to Leduc Hold'em, where the optimum is computable. The result is a lightweight, game-agnostic recipe that trains competitive agents without training on the expert, for any game a small model can handle, reported with robust statistics and released as a reusable package.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。