用自对弈强化学习打造超人类电竞AI，4天训练胜率碾压5000名玩家。
Superhuman AI for Generals.io Using Self-Play Reinforcement Learning
- 基于自对弈强化学习，端到端训练视觉变换器策略。
- 在269场对战中以199胜70负击败顶尖人类选手。
- 自研模拟器速度提升万倍，突破训练瓶颈，适合博弈与策略研究者。
我们提出一个超人类级别的AI代理，用于实时战略游戏Generals.io，该游戏需要长程规划与短时战术应对强不完全信息。在4块NVIDIA H200 GPU上训练四天后，该代理登上超过5000名人类玩家的公开1v1排行榜榜首，领先第二名的差距相当于第二名与第25名之间的差距，并在与两名顶尖人类选手的直接对决中取得269场对战199胜70负的战绩。关键突破在于一款原生JAX的模拟器，在单张GPU上实现每秒数千万帧，相比先前模拟器提速约一万倍。在此基础上，我们通过自对弈训练视觉变换器策略，采用策略梯度循环与稀疏胜负奖励，结合顶优势样本筛选及策略参数指数移动平均。综合结果表明，当快速模拟器消除数据瓶颈后，哪些因素真正重要、哪些不重要变得清晰。
原文摘要 · Abstract (English)
We present a superhuman AI agent for Generals.io, a real-time strategy game that requires both long-horizon planning and short-term tactics under strong imperfect information. Trained for four days on 4x NVIDIA H200 GPUs, our agent reaches #1 on the public 1v1 leaderboard of over 5,000 human players, leading the second-ranked player by the same margin that separates second place from 25th, and beats the two top-ranked humans head-to-head with a combined 199-70 record across 269 ladder matches. A key enabler is a JAX-native simulator that reaches tens of millions of frames per second on a single GPU, roughly a 10,000x speedup over the prior simulator. On top of this, we train a vision transformer policy end-to-end by self-play with a policy-gradient loop and sparse win/loss reward, using top-advantage sample filtering and an exponential moving average of the policy parameters. Taken together, our findings highlight what matters, and what does not, once a fast simulator removes the data bottleneck.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。