arXiv:2609.03660cs.LGcs.AI2026-09

用表格模型玩博弈游戏,让非增量学习方法也能在强化学习中表现优异。

Local Updates, Global Learning (LUGL): Playing Games with non-incremental Learners

  • 将训练拆分为局部更新和全局学习两阶段,解决树模型在强化学习中的分布偏移问题。
  • 在9个博弈任务中,基于LightGBM的模型性能媲美甚至超越DQN和DeepCFR。
  • 证明了非神经网络方法在博弈场景中同样具备竞争力,适合追求高效与可解释性的研究者。

神经网络在强化学习中的主导地位部分源于其增量学习能力,天然适配自对弈训练的在线、非平稳特性。然而,在监督学习中,梯度提升树(如LightGBM)被广泛认为是表格数据的最优解,常比神经网络更准确且高效。游戏状态本质上是表格型的——离散动作、类别牌面、结构化棋盘位置——这使其成为树基方法的理想候选。本文提出LUGL(局部更新,全局学习),一种将数据收集与模型拟合解耦的框架,使LightGBM等非增量学习器能在因分布偏移而原本失效的强化学习环境中运行。LUGL交替执行局部更新阶段(代理进行自对弈并累积表格式更新,如Q值、V值、策略或后悔值)与全局学习阶段(用表格训练函数逼近器以泛化至未见状态,随后重置表格)。我们在四个完全信息博弈(井字棋、连珠、奥赛罗、六角棋)和五个不完全信息博弈(库恩扑克、莱杜克德州扑克、说谎骰子、戈夫斯维尔、翻牌五张德州扑克)上测试该方法,结果表明其性能在所有基准上均达到或超过DQN和DeepCFR。实验表明,学术界对神经网络在博弈中的偏好可能并不合理,基于LightGBM的智能体在全部测试任务中均表现优异。

原文摘要 · Abstract (English)

The dominance of Neural Networks (NNs) in RL is partially due to their incremental learning capability, which naturally suits the online, non-stationary nature of self-play training. However, gradient-boosted trees like LightGBM are widely recognised as the state of the art for tabular data in supervised learning, often outperforming NNs in accuracy and efficiency. Game states are inherently tabular---discrete actions, categorical card identities, structured board positions---which makes them an ideal candidate for tree-based methods. We introduce LUGL (Local Updates, Global Learning), a framework that decouples data collection from model fitting, enabling non-incremental learners such as GBTs to operate in RL settings where they would otherwise fail due to distributional shift. LUGL alternates between a local updates phase, where the agent plays self-play games and accumulates tabular updates (Q-values, V-values, policies, or regret values) in a finite table, and a global learning phase, where the table is used to train a function approximator that generalises to unseen states before the table is reset. We test our approach in four standard perfect-information games (Tic-tac-toe, Connect-4, Othello, and Hex) and five imperfect-information games (Kuhn's poker, Leduc Hold'em, Liar's Dice, Goofspiel, and Flop5 Hold'em), and show that our results are competitive with or superior to DQN and DeepCFR. Our experiments demonstrate that the community's strong bias towards NNs in game-playing may be unwarranted, since LightGBM-based agents achieve competitive or superior performance across all tested benchmarks.

强化学习博弈算法树模型LightGBM

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。