模型训练越深，越能减少最难样本的错误率，且效果可预测。
Learning Shrinks the Hard Tail: Training-Dependent Inference Scaling in a Solvable Linear Model
- 构建潜变量难度模型，模拟不同样本固有难度差异。
- 测试失败率随尝试次数幂律下降，指数随训练量增长并趋于上限。
- 揭示训练能压缩最难样本的误差尾部，适用于模型优化与推理策略设计。
我们分析了一个可解的最后层微调模型中的神经网络缩放规律，其中目标具有内在、实例异质性的难度。在潜变量难度（LID）模型中，每个输入的目标方差由来自重尾分布的潜“精度”决定。尽管泛化损失恢复标准缩放规律，我们的核心贡献在于将其与推理联系起来：通过k次尝试的通过率（pass@k）失败率呈现幂律衰减形式 $k^{-β_ ext{eff}}$，但观测指数 $β_ ext{eff}$ 取决于训练程度。该指数随样本数 $N$ 增大而上升，最终在由难度分布尾部决定的固有极限 $β$ 处饱和。这一耦合表明：学习会缩小误差分布的‘硬尾’——模型泛化误差的提升使 pass@k 曲线更陡峭，直至不可消除的目标方差主导。LID 模型给出了该行为的可验证闭式预测，包括一个计算资源分配规则：应在饱和前优先训练，饱和后增加推理尝试。我们在模拟和两个真实数据代理任务中验证了这些预测：CIFAR-10H（人类标注方差）和数学教师-学生蒸馏任务。
原文摘要 · Abstract (English)
We analyze neural scaling laws in a solvable model of last-layer fine-tuning where targets have intrinsic, instance-heterogeneous difficulty. In our Latent Instance Difficulty (LID) model, each input's target variance is governed by a latent ``precision'' drawn from a heavy-tailed distribution. While generalization loss recovers standard scaling laws, our main contribution connects this to inference. The pass@$k$ failure rate exhibits a power-law decay, $k^{-β_\text{eff}}$, but the observed exponent $β_\text{eff}$ is training-dependent. It grows with sample size $N$ before saturating at an intrinsic limit $β$ set by the difficulty distribution's tail. This coupling reveals that learning shrinks the ``hard tail'' of the error distribution: improvements in the model's generalization error steepen the pass@$k$ curve until irreducible target variance dominates. The LID model yields testable, closed-form predictions for this behavior, including a compute-allocation rule that favors training before saturation and inference attempts after. We validate these predictions in simulations and in two real-data proxies: CIFAR-10H (human-label variance) and a maths teacher-student distillation task.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。