数据筛选比奖励设计更关键,能防止自对弈强化学习崩溃。
Survive or Collapse: The Asymmetric Roles of Data Gating and Reward Grounding in Self-Play RL

- 用数据门控控制任务进入训练池,而非依赖奖励设计。
- 严格的数据门控可让系统稳定,即使奖励无真值信息。
- 发现‘有真值的出题者反而加速崩溃’的反直觉现象。
自对弈强化学习通过模型自动生成任务来训练语言模型,协同进化出出题者与求解者,无需人类标注。近期系统虽展现强推理能力,但普遍出现崩溃与不稳定性,机制尚不明确。现有方法多归因于奖励设计问题,本文提出稳定性由两个独立机制决定:数据级门控(决定哪些任务进入训练)和奖励信号(更新策略)。在去除预训练先验、输出模糊性和执行噪声的确定性DSL任务上,实验表明二者作用不对称:严格门控可保证所有奖励方案下的稳定;而一旦移除门控,任何奖励均无法避免崩溃。该现象揭示‘有真值的出题者反而加速崩溃’的悖论:当与自洽求解器配对时,有真值的出题者会集中训练于干净任务,快速导向虚假自洽吸引子。引入连续门控强度参数ε后,发现训练指标在低ε下即分离,而验证准确率持续至ε更高才下降。结果表明,数据级门控才是自对弈稳定性的瓶颈。
原文摘要 · Abstract (English)
Self-play reinforcement learning trains language models on their own generated tasks, co-evolving a proposer and solver without human labels. Recent systems report strong reasoning gains, but collapse and instability are widely observed and poorly understood. The dominant response treats this as a reward-design problem. We argue instead that self-play stability is governed by two distinct levers: a data-level gate that decides which proposer-generated tasks enter the training pool, and the reward signal that updates the policy on tasks already admitted. Through controlled experiments on a Python output-prediction task and a deterministic-DSL twin task that strips pretraining priors, output ambiguity, and executor noise, we find the two levers are asymmetric. A strict gate is sufficient for stability under every reward variant we test, including a self-consistency reward with no access to ground truth; while no reward variant is sufficient once the gate is removed. This asymmetry exposes a counter-intuitive coupling we call the Grounded Proposer Paradox: a proposer with ground-truth access accelerates collapse faster than an ungrounded one when paired with a self-consistency solver, by concentrating training on clean tasks that form the fastest path to a spurious self-consistent attractor. Replacing the binary gate with a continuous strictness parameter $\varepsilon$ further reveals a two-stage phase transition: training-side metrics decouple at low $\varepsilon$, while validation accuracy holds until $\varepsilon$ is much higher. Data-level gating, not reward calibration, is the binding constraint on self-play stability.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。