通过迭代选择高质量数据，提升大模型指令微调效果
Boosting LLM via Learning from Data Iteratively and Selectively
- 基于复杂度与多样性动态筛选样本，迭代更新评分
- 在多个数据集上超越强基线，提升稳定且显著
- 适用于不同领域和模型，适合需要高效微调的场景
当前数据集通常来自多种来源并使用不同合成技术，因此在后训练前进行去噪和去重至关重要。本文提出一种迭代数据选择方法（IterIT）用于指令微调。该方法同时从复杂度和多样性两个维度衡量样本质量。不同于传统一次性计算复杂度分数，IterIT 在微调过程中动态更新模型相关的复杂度评分，以适应模型状态变化。多样性评分则基于样本响应内容的有用性来定义。IterIT 通过迭代更新高分样本的复杂度，并贪婪选择复杂度-多样性综合得分最高的样本，融合两者优势。在多个指令微调数据集上的实验表明，IterIT 持续优于强基线。此外，该方法在特定领域和不同主干模型上也表现出良好泛化能力。所有资源将公开于 https://github.com/JiaQiSJTU/IterIT。
原文摘要 · Abstract (English)
Datasets nowadays are generally constructed from multiple sources and using different synthetic techniques, making data de-noising and de-duplication crucial before being used for post-training. In this work, we propose to perform instruction tuning by iterative data selection (\ApproachName{}). We measure the quality of a sample from complexity and diversity simultaneously. Instead of calculating the complexity score once for all before fine-tuning, we highlight the importance of updating this model-specific score during fine-tuning to accurately accommodate the dynamic changes of the model. On the other hand, the diversity score is defined on top of the samples' responses under the consideration of their informativeness. IterIT integrates the strengths of both worlds by iteratively updating the complexity score for the top-ranked samples and greedily selecting the ones with the highest complexity-diversity score. Experiments on multiple instruction-tuning data demonstrate consistent improvements of IterIT over strong baselines. Moreover, our approach also generalizes well to domain-specific scenarios and different backbone models. All resources will be available at https://github.com/JiaQiSJTU/IterIT.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。