提出新方法降低指令数据冗余，用3.5%数据超越全量训练
On Representation Redundancy in Large-Scale Instruction Tuning Data Selection
- 通过压缩表示减少语义重复，提升数据选择质量
- 仅用3.5%数据即超越全量基线，平均提升0.71%
- 适合大规模指令微调数据筛选，尤其关注效率的团队
数据质量是大语言模型训练的关键因素。尽管已有研究证明，小而高质量的数据集训练的模型可优于在更大但噪声较多的数据上训练的模型，但在工业级指令微调数据选择方面仍缺乏系统方法。本文从语义表示相似性的角度研究指令微调数据选择，发现当前主流大模型编码器会产生高度冗余的语义嵌入。为此，我们提出压缩表示数据选择（CRDS）框架，包含两种变体：CRDS-R采用Rademacher随机投影并拼接Transformer隐藏层表示；CRDS-W则使用基于白化的方法进行降维以提升表示质量。实验表明，两种变体均显著提升数据质量，并持续优于现有基于表示的选择方法。特别地，CRDS-W仅使用3.5%的数据，就在四个数据集上平均比全量数据基线高出0.71%。代码已开源。
原文摘要 · Abstract (English)
Data quality is a crucial factor in large language models training. While prior work has shown that models trained on smaller, high-quality datasets can outperform those trained on much larger but noisy or low-quality corpora, systematic methods for industrial-scale data selection in instruction tuning remain underexplored. In this work, we study instruction-tuning data selection through the lens of semantic representation similarity and identify a key limitation of state-of-the-art LLM encoders: they produce highly redundant semantic embeddings. To mitigate this redundancy, we propose Compressed Representation Data Selection (CRDS), a novel framework with two variants. CRDS-R applies Rademacher random projection followed by concatenation of transformer hidden-layer representations, while CRDS-W employs whitening-based dimensionality reduction to improve representational quality. Experimental results demonstrate that both variants substantially enhance data quality and consistently outperform state-of-the-art representation-based selection methods. Notably, CRDS-W achieves strong performance using only 3.5% of the data, surpassing the full-data baseline by an average of 0.71% across four datasets. Our code is available at https://github.com/tdano1/CRDS.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。