arXiv:2604.27883math.STcs.IT2026-04

提出新训练方法,让模型训练误差精确跟踪测试误差。

Decoupled Descent: Exact Test Error Tracking Via Approximate Message Passing

论文配图:Decoupled Descent: Exact Test Error Tracking Via Approximate Message Passing
图 1 · 摘自论文原文
  • 基于消息传递理论设计去耦下降算法,消除数据重复使用带来的偏差。
  • 在高斯混合模型下实现训练误差与测试误差渐近一致,支持零成本验证。
  • 适用于小样本和非线性任务,适合追求高泛化性能的研究者。

在现代参数化模型训练中,全批量梯度下降及其变体因对训练数据的过度拟合而产生系统性‘泛化差距’,使训练误差无法可靠反映测试误差。现有方法或通过复杂分析认为该差距可接受,或牺牲数据用于验证集。本文提出去耦下降(DD)——一种基于理论的新训练算法,在特定高斯混合模型下满足训练-测试误差一致性,强制训练误差渐近追踪测试误差。借助近似消息传递理论,DD 迭代消除数据重用导致的偏差,严格证明了零成本验证与100%数据利用率的可行性。此外,DD 由低维状态演化递推公式控制,其动态过程清晰可解析。我们在异或分类任务上验证了DD性能优于标准梯度下降;在带噪MNIST和CIFAR-10非线性探测任务中也显示,即使放松理想假设,DD仍显著缩小泛化差距。

原文摘要 · Abstract (English)

In modern parametric model training, full-batch gradient descent (and its variants) suffers due to progressively stronger biasing towards the exact realization of training data; this drives the systematic ``generalization gap'', where the train error becomes an unreliable proxy for test error. Existing approaches either argue this gap is benign through complex analysis or sacrifice data to a validation set. In contrast, we introduce decoupled descent (DD), a novel theory-based training algorithm that satisfies a train-test identity -- enforcing the train error to asymptotically track the test error for stylized Gaussian mixture models. Within this specific regime, leveraging approximate message passing theory, DD iteratively cancels the biases due to data reuse, rigorously demonstrating the feasibility of zero-cost validation and $100\%$ data utilization. Moreover, DD is governed by a low-dimensional state evolution recursion, rendering the dynamics of the algorithm transparent and tractable. We validate DD on XOR classification, yielding superior performance compared to GD; additionally, we implement noisy MNIST and non-linear probing of CIFAR-10, demonstrating that even when our stylized assumptions are relaxed, DD narrows the generalization gap compared to GD.

训练优化泛化差距消息传递

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。