arXiv:2606.28471cs.AI2026-06

通过能力切片构建评估与数据的闭环,精准定位模型缺陷并针对性优化。

Data and Evaluation Closed-Loop for Model Capability Enhancement

论文配图:Data and Evaluation Closed-Loop for Model Capability Enhancement
图 1 · 摘自论文原文
  • 提出'能力切片'概念,按任务类型和约束聚合评估样本,精准定位弱点。
  • 实验证明可使数学推理成绩从6.67提升至26.67,且修复后性能超越原模型。
  • 适用于模型调优工程师,让数据改进从直觉转向可验证的系统方法。

大语言模型的能力是预训练的核心变量,但无法直接观测:数据前瞻性塑造能力,评估则仅事后揭示,将样本、提示、解码和评分规则压缩为一个噪声分数。实际优化常从失败现象反推,但评估与数据术语不兼容(如基准名与数据源),导致推断依赖直觉而非方法。本文提出‘能力切片’——一组共享背景条件、任务类型、求解操作与输出约束的评估样本,足够精细以定位单一缺陷,又足够稳定支持聚合。基于此构建评估分类、非指令数据分类及映射规则,形成从基准级失败到可测试数据干预的闭环。两个案例验证:其一,持续预训练使BBH下降46.82%,诊断发现根源为单个<eos>损失缺失,修复后恢复至66.44,超原始检查点;其二,数学推理薄弱环节被分解为特定求解组合,针对性采样使AIME2025/AIME2026 Pass@128从6.67/0.00升至26.67。同一闭环在相反方向得出正确结论,证明该推理可常规化、可审计、可实验验证。

原文摘要 · Abstract (English)

Model capability is the central variable in LLM pre-training, yet is never observed directly: data shapes it prospectively, while evaluation reveals it only retrospectively, compressing samples, prompts, decoding, and scoring rules into one noisy score. Practical optimization runs this backward: a failure is observed first, and the engineer must infer the corpus fix. The two sides speak incompatible vocabularies -- benchmark names and per-sample correctness versus data sources, domains, and quality labels -- so this inference is usually intuition, not method. We close this gap with the \emph{capability slice}: a group of evaluation samples sharing background condition, task type, solving operation, and output constraint -- precise enough to localize a single weakness yet stable enough to survive aggregation, unlike a benchmark name, too coarse, or a single sample, too noisy. Built around this unit, an evaluation taxonomy, a non-instruction data taxonomy, and mapping rules form a closed loop turning a benchmark-level failure into a targeted, testable data intervention. We test this loop on two case studies pulling in opposite directions. First, the loop rules the data out: continued pre-training drives BBH down by $-46.82\%$, but diagnosis traces this to a single masked \texttt{\textless EOS\textgreater} loss rather than weakened reasoning; restoring it recovers BBH to $66.44$, above the original checkpoint, without changing the data. Second, the loop rules the data in: a persistent math-reasoning weakness is decomposed by solving operation into specific failing combinations, and a weakness-targeted sampling procedure built from it lifts AIME2025/AIME2026 Pass@128 from $6.67$/$0.00$ to $26.67$ each. The same unmodified loop reaches opposite, correct verdicts in both cases, showing the evaluation-to-data inference can be routine, auditable, and experimentally validated rather than intuitive.

模型优化评估闭环能力切片数据工程

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。