MIRA通过自锚定规则发现，实现源感知的数据筛选，提升大模型训练效率。
MIRA: Mid-training Rubric Anchoring for Source-Aware Data Selection
- 基于自锚定规则发现，动态为不同数据源构建评估标准。
- 在21个源、5组数据上，仅用一半token达到全量训练效果。
- 适合需要高效、源适应性数据筛选的大模型中段训练场景。
中段训练已成为现代大模型开发的关键阶段，使用大规模精调混合数据以增强能力，再进行最终后训练。其数据选择问题具有独特性：数据在接近预训练规模下，以预训练式目标优化，但需针对下游任务能力进行筛选，且来源多样、格式和训练角色各异。因此，有效筛选需兼顾可扩展性与源自适应的语义标准。现有基于模型的方法虽可扩展，但仅提供隐式质量信号；语义筛选方法虽判断更强，却通常依赖固定规则或标准化格式。为此，我们提出MIRA，一种基于自锚定规则发现的源感知过滤框架。核心思路是将规则构建纳入数据筛选过程：MIRA先为每类数据源发现应评估的内容，再将其判断提炼为可扩展的学生评分器，用于全数据集过滤。在包含21个源、5个源组的代码导向中段训练中，MIRA在9个代码基准上优于基线方法，并在仅使用一半令牌的情况下达到全量训练性能。
原文摘要 · Abstract (English)
Mid-training has become an important stage in modern LLM development, using large-scale curated mixtures to strengthen capabilities before final post-training. Its data selection problem is distinct: the data are optimized under a pretraining-style objective at near-pretraining scale, but are curated toward downstream capabilities and drawn from heterogeneous sources with different formats and training roles. As a result, effective selection requires both scalability and source-adaptive semantic criteria. Existing model-based methods scale well, but provide only implicit quality signals. Semantic selection methods offer stronger judgments, but usually assume fixed rubrics or standardized data formats. To address this mismatch, we propose MIRA, a source-aware filtering framework based on self-anchored rubric discovery. The key idea is to make rubric construction part of data selection: MIRA first discovers what should be evaluated for each source group, then distills those judgments into scalable student scorers for full-corpus filtering. On code-oriented mid-training with 21 sources and 5 source groups, MIRA outperforms selection baselines across nine code benchmarks and matches the full-corpus run while using only half the tokens.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。