CRAFT通过聚类回归实现训练数据自适应筛选,提升翻译模型性能并大幅提速。
CRAFT: Clustered Regression for Adaptive Filtering of Training data

- 基于聚类与条件距离最小化,分两阶段筛选高质量训练数据
- 在3300万句对上达到43.34 BLEU,比TSDS高2.13点,速度超40倍
- 支持任意向量化方式,CPU上全流程<1分钟,适合大规模数据精调
随着语料库规模增至数千万条数据,从其中选取少量高质量子集进行微调变得愈发重要,全量微调成本过高且常非必要。本文提出CRAFT(Clustered Regression for Adaptive Filtering of Training data),一种无需依赖向量化方法的序列到序列模型训练数据选择技术。CRAFT将源-目标联合分布分解,采用两阶段筛选策略:(i) 通过按比例分配预算至k-means聚类,匹配验证集的源分布;(ii) 在每个源聚类内,选择目标嵌入使条件期望距离最小化的训练样本对,该距离基于验证集目标分布推导。理论证明,比例聚类分配可控制选定数据与验证分布之间的连续KL散度,残差由聚类直径决定。在英印翻译任务中,从3300万条NLLB句对中筛选数据,并使用LoRA微调mBART。CRAFT取得43.34 BLEU,较相同候选池和编码器的TSDS(41.21)高出2.13点,且筛选速度超过40倍。采用TF-IDF向量化时,整个流程可在CPU上不到1分钟完成。虽然TAROT达到45.61 BLEU,但CRAFT仅需26.86秒完成筛选,相比TAROT的75.6秒提速2.8倍。
原文摘要 · Abstract (English)
Selecting a small, high-quality subset from a large corpus for fine-tuning is increasingly important as corpora grow to tens of millions of datapoints, making full fine-tuning expensive and often unnecessary. We propose CRAFT (Clustered Regression for Adaptive Filtering of Training data), a vectorization-agnostic selection method for training sequence-to-sequence models. CRAFT decomposes the joint source-target distribution and performs a two-stage selection: (i) match the validation source distribution through proportional budget allocation across k-means clusters, and (ii) within each source cluster, select training pairs whose target embeddings minimize a conditional expected distance derived from the validation target distribution. We prove that proportional cluster allocation bounds the continuous KL divergence between selected and validation distributions, with the residual controlled by cluster diameters. We evaluate CRAFT on English-Hindi translation by selecting training data from 33 million NLLB sentence pairs and fine-tuning mBART via LoRA. CRAFT achieves 43.34 BLEU, outperforming TSDS (41.21) by 2.13 points on the same candidate pool and encoder while completing selection over 40 times faster. With TF-IDF vectorization, the entire pipeline completes in under one minute on CPU. TAROT achieves 45.61 BLEU, but CRAFT completes selection in 26.86 seconds versus TAROT's 75.6 seconds, a 2.8 time speedup.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。