GRIP通过几何建模提升数据效率，让小数据集表现媲美三倍大的未清洗数据集。
GRIP: Geometric Refinement and Adaptive Information Potential for Data Efficiency
- 将语料库视为信息密集的几何空间，统一全局分布与局部选择。
- 在300B tokens的MoE模型上，用更少数据超越三倍体量未清洗数据的效果。
- 适合追求高效训练的大模型研究者和工业级预训练团队。
大型语言模型（LLMs）的性能越来越受数据效率而非单纯规模驱动。现有选择方法常将全局分布平衡与局部实例选择割裂，破坏训练集的层次结构。我们提出 extbf{GRIP}（几何精炼与自适应信息势），将语料库建模为信息密集的几何空间，通过 extbf{快速适应探针（RAP）}量化语义簇的信息潜力，动态向代表性不足区域分配采样预算。随后，采用 extbf{长度修正的几何先验}进行簇内选择，以抵消嵌入密度偏差并保留长尾逻辑序列。在高达300B token的Mixture-of-Experts（MoE）模型上广泛评估表明，GRIP持续优于现有最佳基线， extbf{性能超过使用三倍大且未经清理数据训练的模型}。本工作为大规模预训练中的自适应数据精炼建立了稳健的几何基础。
原文摘要 · Abstract (English)
The performance of Large Language Models (LLMs) is increasingly governed by data efficiency rather than raw scaling volume. However, existing selection methods often decouple global distribution balancing from local instance selection, compromising the hierarchical integrity of the training set. We introduce \textbf{GRIP} (Geometric Refinement and Adaptive Information Potential), a framework that unifies these dimensions by modeling the corpus as an information-dense geometric space. GRIP employs a \textbf{Rapid Adaptation Probe (RAP)} to quantify the information potential of semantic clusters, dynamically re-allocating the sampling budget to regions with the highest representation deficits. Subsequently, we perform Intra-Cluster Selection using a \textbf{length-rectified geometric prior} to counteract embedding density artifacts and preserve long-tail logical sequences. Extensive evaluations on Mixture-of-Experts (MoE) models up to 300B tokens demonstrate that GRIP consistently outperforms state-of-the-art baselines, \textbf{surpassing the performance of models trained on $3\times$ larger uncurated datasets}. Our work establishes a robust geometric foundation for adaptive data curation in large-scale pre-training.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。