通过权重插值提升语言模型推理能力，缓解生成多样性下降问题
Weight Ensembling Improves Reasoning in Language Models
- 用最新与早期微调权重插值，改善推理模型生成多样性
- 在数学推理任务中，Pass@1和Pass@k均显著提升，尤其在少数据场景下表现更好
- 适用于追求高推理准确率的模型优化，对强化学习微调有额外增益
我们研究了推理模型训练中的一个失效现象：生成多样性逐渐崩溃，导致测试时扩展效果不佳。值得注意的是，经过监督微调（SFT）后，Pass@1指标持续提升，但Pass@k迅速恶化。令人意外的是，将最新SFT检查点与早期检查点的权重进行插值（即WiSE-FT），几乎完全恢复了Pass@k性能，同时进一步提升了Pass@1。该方法在测试时扩展（Best@k、多数投票）上表现更优，且在后续强化学习微调中以更少数据取得更佳结果。此外，发现WiSE-FT带来的增益无法仅通过温度调节等多样性解码策略实现。我们形式化了Pass@k在测试分布上的偏差-方差权衡，发现WiSE-FT可同时降低偏差与方差，而温度调节则固有地在两者间权衡。
原文摘要 · Abstract (English)
We investigate a failure mode that arises during the training of reasoning models, where the diversity of generations begins to collapse, leading to suboptimal test-time scaling. Notably, the Pass@1 rate reliably improves during supervised finetuning (SFT), but Pass@k rapidly deteriorates. Surprisingly, a simple intervention of interpolating the weights of the latest SFT checkpoint with an early checkpoint, otherwise known as WiSE-FT, almost completely recovers Pass@k while also improving Pass@1. The WiSE-FT variant achieves better test-time scaling (Best@k, majority vote) and achieves superior results with less data when tuned further by reinforcement learning. Finally, we find that WiSE-FT provides complementary performance gains that cannot be achieved only through diversity-inducing decoding strategies, like temperature scaling. We formalize a bias-variance tradeoff of Pass@k with respect to the expectation and variance of Pass@1 over the test distribution. We find that WiSE-FT can reduce bias and variance simultaneously, while temperature scaling inherently trades off between bias and variance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。