用奖励重加权负样本，让大模型更高效地从自生成数据中学习。
RIFT: Repurposing Negative Samples via Reward-Informed Fine-Tuning
- 通过奖励重加权，让负样本也能参与训练
- 在多个数学基准上优于传统拒绝采样微调方法
- 适合缺乏专家数据、想提升数据利用效率的场景
监督微调（SFT）和拒绝采样微调（RFT）是大模型对齐的标准方法，但前者依赖昂贵的专家数据，后者会丢弃有价值的负样本，导致数据利用效率低下。为此，我们提出奖励感知微调（RIFT），一种简单而有效的框架，可充分利用所有自生成样本。与RFT的硬阈值筛选不同，RIFT重新利用负轨迹，通过标量奖励对损失函数进行重加权，从而同时从正负轨迹中学习。为克服直接奖励融合引发的训练崩溃问题（即损失无界），我们引入稳定化损失形式，确保数值鲁棒性和优化效率。在多种基础模型上的数学基准测试表明，RIFT始终优于RFT。结果证明，RIFT是一种鲁棒且数据高效的对齐方法，适用于混合质量的自生成数据。
原文摘要 · Abstract (English)
While Supervised Fine-Tuning (SFT) and Rejection Sampling Fine-Tuning (RFT) are standard for LLM alignment, they either rely on costly expert data or discard valuable negative samples, leading to data inefficiency. To address this, we propose Reward Informed Fine-Tuning (RIFT), a simple yet effective framework that utilizes all self-generated samples. Unlike the hard thresholding of RFT, RIFT repurposes negative trajectories, reweighting the loss with scalar rewards to learn from both the positive and negative trajectories from the model outputs. To overcome the training collapse caused by naive reward integration, where direct multiplication yields an unbounded loss, we introduce a stabilized loss formulation that ensures numerical robustness and optimization efficiency. Extensive experiments on mathematical benchmarks across various base models show that RIFT consistently outperforms RFT. Our results demonstrate that RIFT is a robust and data-efficient alternative for alignment using mixed-quality, self-generated data.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。