根据目标模型能力选数据，提升微调效率和效果
Data-DPO: Direct Preference Optimization for Target Model Data Selection in LLM Post-Training
- 通过探测模型对不同数据的反馈，构建数据偏好关系
- 在多种数据预算下表现优于现有方法，甚至超越全量数据训练
- 适合需要高效微调大模型的研究者与工程师
监督微调中的数据选择旨在从大规模候选数据中挑选少量有效样本，以降低训练成本并保持模型性能。然而，现有方法通常将数据价值视为相对静态属性，且较少关注数据与目标模型能力分布的适配性。为此，我们提出 Data-DPO，一种面向目标模型的 SFT 数据选择方法。Data-DPO 通过单步探测获取目标模型对不同样本的局部训练反馈，将样本间的激活差异转化为成对数据偏好，并训练一个轻量级奖励模型以学习目标模型感知的数据偏好。在最终选择阶段，Data-DPO 进一步结合目标模型偏好、外部质量评分与边际多样性，构建更稳定高效的训练子集。在 Vision-Flan 与 LLaVA-CoT 上的实验表明，Data-DPO 在多种数据预算下均持续优于现有基线，且稳定超越全量数据训练性能。
原文摘要 · Abstract (English)
Data selection in supervised fine-tuning aims to select a small set of effective samples from large-scale candidate data, reducing training cost while preserving model performance. However, existing methods usually treat data value as a relatively static property, and pay limited attention to the compatibility between data and the capability distribution of the target model. To address this issue, we propose Data-DPO, a target model-oriented SFT data selection method. Data-DPO observes the local training feedback of the target model on different samples through one-step probing, transforms activation differences among samples into pairwise data preferences, and trains a lightweight reward model to learn target-model-aware data preferences. In the final selection stage, Data-DPO further combines target model preference, external quality scores, and marginal diversity to construct a more stable and effective training subset. Experimental results on Vision-Flan and LLaVA-CoT show that Data-DPO consistently outperforms existing data selection baselines under multiple data budgets and stably surpasses full data training performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。