用智能编辑提升大模型训练数据质量，效率更高更可靠。
UltraX: Refining Pre-Training Data at Scale with Adaptive Programmatic Editing
- 引入插入、删除、修改三类操作，实现细粒度数据修正。
- 通过程序化监督与动态采样，减少低质量样本影响。
- 适合大规模数据清洗，尤其适合追求数据效率的团队。
随着可用训练数据接近物理极限，规模定律带来的收益逐渐减弱。因此，当前大型语言模型（LLMs）的提升不再依赖数据量扩大，而更多取决于高质量数据的利用。然而，在大规模语料库中，现有数据精炼方法在质量、效率和可靠性方面存在显著局限：基于规则的方法受固定启发式限制，难以应对实例级差异；基于大模型的方法虽能提升质量，但无法满足大规模处理所需的效率与可靠性。为此，我们提出UltraX，一种基于函数调用的大规模预训练数据精炼框架，通过引入插入操作，补全了编辑功能空间，实现了细粒度的实例级编辑。具体而言，UltraX构建了一个可靠的程序监督生成流水线：首先，通过数据集自适应提示优化，引导专家大模型生成高质量端到端精炼文本；随后，利用行对齐映射与动态上下文替换，将原始-精炼文本对转化为结构化程序监督。同时，通过低置信度样本过滤与操作组合比例控制采样，提升监督质量并稳定训练分布。推理与执行阶段，通过滑动窗口预测、全局操作聚合与系统性后处理，对模型输出进行归一化与验证，显著提升大规模执行的稳定性与可靠性。实验表明，UltraX在所有语料库上均达到最高平均性能，且使用更少训练标记即匹配或超越基线，展现出更强的数据效率与精炼可靠性。
原文摘要 · Abstract (English)
As available training data approaches its physical limit, gains from Scaling Laws have begun to diminish. Consequently, improving Large Language Models (LLMs) now depends less on data expansion and more on higher-quality data utilization. However, in the context of large-scale corpora, existing refinement methodologies face significant limitations in quality, efficiency, and reliability: Rule-based approaches are constrained by fixed heuristics and struggle with instance-level variations; LLM-based approaches improve quality but fail to meet the efficiency and reliability requirements of large-scale data processing. To address these challenges, we propose UltraX, a function-calling refinement framework for large-scale pre-training data that completes the editing function space by introducing insertion in addition to deletion and modification, enabling fine-grained instance-level editing. Specifically, UltraX builds a reliable program-supervision generation pipeline. In this pipeline, dataset-adaptive prompt optimization first guides an expert LLM to produce high-quality end-to-end refined texts, and Line Alignment Mapping and Dynamic Context Replacement then convert original-refined text pairs into structured program supervision. Meanwhile, UltraX improves supervision quality and stabilizes the training distribution with low-confidence example filtering and ratio-controlled sampling by operation combination. During inference and execution, it normalizes and validates model outputs through sliding-window prediction, global operation aggregation, and systematic post-processing, improving the stability and reliability of large-scale execution. Experiments show that UltraX achieves the highest average performance across all corpora and also matches or surpasses baselines with fewer training tokens, demonstrating stronger data efficiency and refinement reliability.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。