统一数据混合与筛选，提升大模型训练效率和质量
UniGeM: Unifying Data Mixing and Selection via Geometric Exploration and Mining
- 将数据整理视为流形逼近问题，分层处理混合权重与高质量样本筛选
- 在1000亿token上训练80亿和160亿参数模型，数据效率提升2倍
- 适合关注大模型数据质量、代码语料结构保持的研究者
大语言模型的扩展日益受限于数据质量。现有方法通常将数据混合与样本筛选分开处理，可能破坏代码语料中的结构。我们提出UniGeM框架，通过将数据整理视为流形逼近问题，无需训练代理模型或依赖外部参考数据集，实现混合与筛选的统一。该框架分层运作：宏观探索通过基于稳定性的聚类学习混合权重；微观挖掘则根据几何分布筛选高质量实例，确保逻辑一致性。在1000亿个标记上训练80亿和160亿参数的MoE模型验证表明，UniGeM相比随机基线实现2.0倍的数据效率，并在推理密集型评估和多语言泛化任务中超越当前最优方法。
原文摘要 · Abstract (English)
The scaling of Large Language Models (LLMs) is increasingly limited by data quality. Most methods handle data mixing and sample selection separately, which can break the structure in code corpora. We introduce \textbf{UniGeM}, a framework that unifies mixing and selection by treating data curation as a \textit{manifold approximation} problem without training proxy models or relying on external reference datasets. UniGeM operates hierarchically: \textbf{Macro-Exploration} learns mixing weights with stability-based clustering; \textbf{Micro-Mining} filters high-quality instances by their geometric distribution to ensure logical consistency. Validated by training 8B and 16B MoE models on 100B tokens, UniGeM achieves \textbf{2.0$\times$ data efficiency} over a random baseline and further improves overall performance compared to SOTA methods in reasoning-heavy evaluations and multilingual generalization.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。