破解数学强化学习中的难度断层问题，提升求解效率。
The Two-Hump Problem: Bridging the Difficulty Gap in Mathematical Reinforcement Learning
- 构建新数据生成方法填补难题空白区
- 引入超移动与Transformer架构显著提效
- 发布首个大规模公开数学难题数据集
数学搜索问题因庞大的搜索空间和稀疏奖励，对强化学习构成独特挑战。以往研究以Andrews-Curtis（AC）猜想为例，本文发现其问题结构存在‘双峰’分布：问题要么极易解决，要么几乎不可解，缺乏中间难度的可解实例，阻碍有效学习。为此，我们提出两种解决方案：创新的数据生成技术填补难度断层，以及算法层面的重大改进，包括引入超移动（supermoves）和基于Transformer的架构。实验表明性能显著优于基线模型，并发布了两个全新基准数据集：AC-19（包含125,192个长度不超过19的平凡可解实例）和AC-1M（包含1,136,154个长度不超过30的困难但可解实例），这是首个大规模、公开可用的同类数据集。
原文摘要 · Abstract (English)
Mathematical search problems present a unique challenge for Reinforcement Learning (RL) due to vast search spaces and sparse rewards. In previous works, the Andrews-Curtis (AC) conjecture was established as an illustrative example of such problems. In this work, we identify a critical structural barrier in the AC landscape: a "Two-Hump" distribution, where problem instances are either trivially solvable or effectively impossible, with a scarcity of intermediate "hard-but-solvable" instances required for effective learning. We tackle this challenge through two primary avenues: novel data generation techniques to populate the difficulty gap, and significant algorithmic enhancements including the introduction of supermoves and Transformer-based architectures. We demonstrate substantial performance improvements over previous baselines, and release new comprehensive benchmark datasets including AC-19 (125,192 AC-trivial presentations of varying difficulty with length at most 19) and AC-1M (1,136,154 hard AC-trivial presentations of length at most 30), the first large-scale, publicly available datasets of this kind.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。