arXiv:2608.20290cs.AIcs.CL2026-08被引 2

发现自提升评估中七个测量陷阱,揭示多数结论可能是假象。

Phantom Gains: Auditing Self-Improvement Against a Measured Null

论文配图:Phantom Gains: Auditing Self-Improvement Against a Measured Null
图 1 · 摘自论文原文
  • 用冻结对照组对比三次自训练,识别出七种测量错误
  • 自训练看似提升,实为批处理等误差造成的假象,真实无增益
  • 提出基于池化基线的精确检验法,适用于多臂实验复现

在三轮针对 Qwen3-8B 的秩-32 LoRA 自训练审计中,与冻结对照组通过相同流程比较,发现七种测量缺陷,每种均会因缺少对照而反转原有结论。例如,仅用贪婪解码构建的记录表会在未训练模型上制造出能力提升的假象,其扩展统计量显示率为0.280;自然阈值修复法在复现中不成立,其零假设在冻结对照中仍非零。本文改用每问题精确检验结合虚假发现率控制的池化基线方法,在所有保留样本中均未检测到显著变化,且不受多重检验规则、误报率和池大小影响。应用于匹配流、数量与评估的多臂阶梯实验发现,外部蒸馏可改善基础模型极少触及的问题,但三种自训练方式均未带来实际提升;回归分析显示该差异源于蒸馏整体增益(p < 10⁻⁸)。对于基础模型从未解决的问题,证据不足;而自训练反而以远超测量下限的速率破坏已解决的问题。因此,过渡层面审计需为每项统计量单独建立测量零假设,这些零假设可基于多臂研究已有重复实验构建,无需额外实验。

原文摘要 · Abstract (English)

Whether a language model has improved itself is increasingly judged not by mean accuracy but by which individual problems it gains and loses. Tracking these transitions means differencing two noisy estimates, leaving them vulnerable to measurement artifacts. Auditing three rounds of rank-$32$ LoRA self-training on Qwen3-8B against a frozen control pushed through the identical pipeline, we identify seven measurement failures, each of which inverts a reported finding when its control is absent. Several are standard practice. A ledger built on a single greedy decode manufactures capability changes on an untrained model, largely an artifact of inference batching; the expansion statistic separating acquisition from sharpening assigns that same model a rate of $0.280$. The natural threshold repair does not survive replication: estimated across the frozen comparisons such a design already contains, its null stays non-zero. We replace it with a per-problem exact test against a pooled baseline under false-discovery-rate control, which detects nothing on any held-out replicate and is unchanged under the multiple-testing rule, error rate and pool size. Applied to a ladder of arms matched in stream, volume and evaluation, the audit finds that external distillation improves problems the base model rarely reaches while three forms of self-training do not; a regression rejects this asymmetry as a by-product of distillation's larger overall gain ($p < 10^{-8}$). On the far smaller set of problems the base model never reaches, the evidence is inconclusive, while self-training corrupts problems solved at baseline at rates well above the measured floor. Transition-level auditing therefore requires a separately measured null for every statistic it reports: nulls that cost no new experiments, built from baseline replicates a multi-arm study already owns, though not from as few as most possess.

模型评估自训练审计方法

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。