通过协同缩放数据与计算，提升大模型推理训练效率与稳定性。
CoScale-RL: Efficient Post-Training by Co-Scaling Data and Computation
- 收集多解增强问题可解性，替代简单扩增数据集。
- 滚动计算缩放使强化学习更稳定，平均准确率提升3.76倍。
- 融合重蒸馏技术，实现高效扩展且不增加计算负担。
训练大型推理模型（LRM）通常不稳定且难以预测，尤其在难题或基础模型较弱时。我们发现现有后训练缩放策略仍可改善此类情况。本文提出CoScale-RL，一种新的缩放策略，兼具更高数据与计算效率。首先通过收集每个问题的多个解来扩大解决方案规模，使难题更具可解性；其次通过扩大滚动计算量来稳定强化学习过程。进一步采用名为重蒸馏（Re-distillation）的模型融合技术，在缩放过程中保持甚至提升计算效率。该方法在四个基准测试上平均实现3.76倍的准确率提升，无需大量监督微调数据即可突破大模型能力边界。CoScale-RL为提升大模型推理能力提供了新的缩放方向。
原文摘要 · Abstract (English)
Training Large Reasoning Model (LRM) is usually unstable and unpredictable, especially on hard problems or weak foundation models. We found that the current post-training scaling strategy can still improve on these cases. We propose CoScale-RL, a novel scaling strategy with better data and computational efficiency. We first scale up solutions to make problems solvable. The core idea is to collect multiple solutions for each problem, rather than simply enlarging the dataset. Then, we scale up rollout computation to stabilize Reinforcement Learning. We further leverage a model merge technique called Re-distillation to sustain or even improve computational efficiency when scaling up. Our method significantly improves data and computational efficiency, with an average 3.76$\times$ accuracy improvement on four benchmarks. CoScale-RL is able to improve an LRM's ability boundary without an extensive SFT dataset. Our method provides a new scaling direction to further improve LRM's reasoning ability.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。