直接优化数据多样性，提升预训练数据质量。
Spokes: Optimizing for Diverse Pretraining Data Selection
- 基于G-Vendi分数，用指数梯度下降直接优化数据集多样性。
- 在50万样本上多样性得分提升489，下游任务性能最高提升1.5点。
- 适合需要高质量、低冗余数据的模型预训练场景。
多样性在数据选择中至关重要，能在固定数据预算下通过减少冗余和重复来提升性能。然而，多样性是全局属性，依赖数据点间的交互关系，难以直接优化。现有方法多依赖代理或近似，常无法保证足够多样性的子集。本文提出一种基于G-Vendi分数的概率化多样性优化框架，采用指数梯度下降进行优化。该方法生成的数据子集显著优于随机采样，在50万样本子集上G-Vendi分数提升489。在FineWeb和DCLM数据集上评估显示，SPOKES（仅优化多样性）分别比随机采样在下游任务上提升0.4和0.5分；同时优化质量和多样性时，性能进一步提升至+1.5和+1.4分，优于所有基线方法，包括语义去重和质量过滤。
原文摘要 · Abstract (English)
Diversity plays a critical role in data selection, improving performance under fixed data budgets by reducing redundancy and repetition. However, optimizing for diversity is inherently challenging, as it is a set-level property that depends on interactions between data points rather than individual examples. As a result, existing approaches typically rely on proxies or approximations, which often fail to ensure sufficiently diverse subsets. In this work, we directly optimize diversity by introducing a probabilistic diversification framework based on the G-Vendi score, optimized via exponentiated gradient descent. Our method produces subsets that are substantially more diverse than those obtained via random sampling, achieving a +489 increase in G-Vendi score on a 500k-sample subset. We evaluate our approach on FineWeb and DCLM, where it consistently outperforms existing methods. Notably, SPOKES (diversity-only) improves average downstream performance by +0.4 and +0.5 points over random sampling on DCLM and FineWeb, respectively. More importantly, jointly optimizing for both quality and diversity yields the strongest results: SPOKES achieves gains of +1.5 and +1.4 points on DCLM and FineWeb, outperforming all baselines, including semantic deduplication and quality filtering.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。