发现强化学习中状态频率服从幂律分布，解释了模型规模与性能的关系。
AlphaZero Neural Scaling and Zipf's Law: a Tale of Board Games and Power Laws
- 基于语言模型的量子化缩放理论，分析AlphaZero训练数据中的状态频率分布。
- 状态频率符合齐普夫定律，且高频状态（如终局）在大模型中占比异常上升。
- 大模型过度关注不重要的终局状态，导致对早期关键状态理解下降。
神经网络缩放规律在多个领域被观察到，但其普遍成因尚不明确。近期理论指出，损失的幂律关系源于齐普夫定律，该定律在自然语言等场景中普遍存在。有理论认为，当按频率降序学习齐普夫分布的任务单元时，会形成语言缩放规律。本文以语言模型缩放模型为基础，研究AlphaZero中的幂律缩放现象。发现训练与推理数据中的游戏状态频率符合齐普夫定律，这源于环境的树状结构。我们验证了状态损失优化确实按频率降序进行，尽管高频状态的建模复杂度反而更低。此外，反向缩放（模型增大后性能不升反降）与异常齐普夫曲线相关：终局状态成为最频繁状态。我们发现，大模型更聚焦于这些次要状态，牺牲了对重要开局状态的理解。
原文摘要 · Abstract (English)
Neural scaling laws are observed in a range of domains, to date with no universal understanding of why they occur. Recent theories suggest that loss power laws arise from Zipf's law, a power law observed in domains like natural language. One theory suggests that language scaling laws emerge when Zipf-distributed task quanta are learned in descending order of frequency. In this paper we examine power-law scaling in AlphaZero, a reinforcement learning algorithm, using a model of language-model scaling. We find that game states in training and inference data scale with Zipf's law, which is known to arise from the tree structure of the environment, and examine the correlation between scaling-law and Zipf's-law exponents. In agreement with the quanta scaling model, we find that agents optimize state loss in descending order of frequency, even though this order scales inversely with modelling complexity. We also find that inverse scaling, the failure of models to improve with size, is correlated with unusual Zipf curves where end-game states are among the most frequent states. We show evidence that larger models shift their focus to these less-important states, sacrificing their understanding of important early-game states.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。