随机稀疏子网络在强化学习中表现媲美全模型，揭示多重可训练路径。
The Multiple Ticket Hypothesis: Random Sparse Subnetworks Suffice for RLVR
- 仅训练1%参数，随机选择亦能达成全模型效果。
- 不同随机掩码成功率高，重叠率低于0.005，说明存在多条有效路径。
- 适用于想压缩模型或加速训练的研究者，尤其适合强化学习场景。
彩票理论假设表明，稀疏子网络可达到全模型性能，暗示参数冗余。在可验证奖励的强化学习（RLVR）中，近期研究发现更新集中在少数参数上，进一步支持这一冗余现象。本文探索最简单的利用方式：极端稀疏下仅随机训练部分参数。实验显示，仅训练1%参数即可在3个模型和2个任务领域中匹配或超越全参数微调性能。不同随机掩码间重叠极低（Jaccard相似度≤0.005），但均成功，表明预训练模型中存在多个可行稀疏子网络，而非单一最优解。我们提出‘多重彩票假设’。通过分析RLVR中的每步隐式KL约束，解释其将更新限制在低维子空间，从而允许任意稀疏掩码成功。
原文摘要 · Abstract (English)
The Lottery Ticket Hypothesis demonstrated that sparse subnetworks can match full-model performance, suggesting parameter redundancy. Meanwhile, in Reinforcement Learning with Verifiable Rewards (RLVR), recent work has shown that updates concentrate on a sparse subset of parameters, which further lends evidence to this underlying redundancy. We study the simplest possible way to exploit this redundancy: training only a randomly selected subset of parameters at extreme sparsities. Empirically, we find that training just 1\% of parameters matches or exceeds full-parameter RLVR finetuning across 3 models and 2 task domains. Moreover, different random masks show minimal overlap ($\leq 0.005$ Jaccard similarity) and yet all succeed, suggesting pretrained models contain many viable sparse subnetworks rather than one privileged set. We term this the Multiple Ticket Hypothesis. We explain this phenomenon through the implicit per-step KL constraint in RLVR, which restricts updates to a low-dimensional subspace, enabling arbitrary sparse masks to succeed.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。