动态筛选关键训练样本，提升大模型推理能力的微调效果
Dynamic Important Example Mining for Reinforcement Finetuning
- 基于梯度对齐估计样本重要性，实时调整数据权重
- 在多个推理基准上超越静态与动态基线方法
- 适合需要高效微调推理能力的研究者和工程师
强化微调（RFT）被广泛用于提升大模型的推理能力，但其效果受限于训练数据的选择与使用方式。现有以数据为中心的RFT方法多依赖静态或启发式样本选择，隐含假设样本价值在训练过程中恒定，忽略了策略学习中的非平稳动态，可能导致优化效果不佳。本文提出动态重要样本挖掘（DIEM），一个原理严谨且完全自动化的框架，使数据利用在RFT全程中保持自适应。DIEM在每一步优化中集成两个组件：(i) 基于梯度对齐的重要性估计算法，高效近似每个样本对策略改进的边际贡献；(ii) 受限的批次重加权机制，在最大化整体效用的同时保持更新梯度幅值，以稳定优化过程。在多个推理基准测试中，DIEM始终优于强基线方法。代码将通过 https://github.com/hrtan/DIEM 发布。
原文摘要 · Abstract (English)
Reinforcement fine-tuning (RFT) is increasingly used to strengthen the reasoning abilities of large models, yet its effectiveness is bound by how training data are selected and used. Most data-centric RFT methods rely on static or heuristic sample selection, implicitly assuming a sample's value is fixed over training. This overlooks the non-stationary dynamics of policy learning and can lead to suboptimal updates. We propose Dynamic Important Example Mining (DIEM), a principled and fully automated framework that makes data utilization adaptive throughout RFT. DIEM integrates two components into each optimization step: (i) a gradient-alignment importance estimator that efficiently approximates each sample's marginal contribution to policy improvement; and (ii) a constrained batch reweighting scheme that maximizes aggregate utility while preserving the update's gradient magnitude to stabilize optimization. Across several reasoning benchmarks, DIEM consistently outperforms strong static and dynamic baselines. The code will be released via https://github.com/hrtan/DIEM.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。