提出可训练的神经网络统一逼近方法,适配带噪数据且参数高效。
Beyond Universal Approximation Theorems: Algorithmic Uniform Approximation by Neural Networks Trained with Noisy Data
- 设计特定架构的随机训练算法,从带噪样本构造统一逼近器。
- 参数量达极小化(对数因子可忽略),接近理论最优。
- 复现真实模型特性:小参数微调、过拟合训练集、合理光滑性。
机器学习的核心目标是训练出能从带噪观测中可靠泛化的模型;然而,当前主流通用逼近定理(UAT)在无噪数据和自由参数选择的理想假设下运作,与实际脱节。本文引入一种针对特定架构的随机训练算法,在d维立方体[0,1]^d上,仅用N个带噪样本即可构造出统一逼近器。所训练的神经网络在可训练参数量上达到极小化(对数因子可忽略),符合经典UAT理想噪声情形下的最小化要求。此外,模型复现了真实神经网络的关键行为:(1)在结构相关且有利的分布外任务上呈现次线性参数复杂度;(2)精确插值训练数据;(3)经初始聚类注意力层后保持合理的利普希茨正则性。这些性质使先进UAT更贴近实际应用,将核心开放问题从‘算法在带噪样本下是否可实现’转变为‘随机梯度下降能否获得类似保证’。
原文摘要 · Abstract (English)
At its core, machine learning seeks to train models that reliably generalize beyond noisy observations; however, the theoretical vacuum in which state-of-the-art universal approximation theorems (UATs) operate isolates them from this goal, as they assume noiseless data and allow network parameters to be chosen freely, independent of algorithmic realism. This paper bridges that gap by introducing an architecture-specific randomized training algorithm that constructs a uniform approximator from $N$ noisy training samples on the $d$-dimensional cube $[0,1]^d$. Our trained neural networks attain the minimax-optimal quantity of \textit{trainable} (non-random) parameters, subject to logarithmic factors which vanish under the idealized noiseless sampling assumed in classical UATs. Additionally, our trained models replicate key behaviours of real-world neural networks, absent in standard UAT constructions, by: (1) exhibiting sub-linear parametric complexity when fine-tuning on structurally related and favourable out-of-distribution tasks, (2) exactly interpolating the training data, and (3) maintaining reasonable Lipschitz regularity (after the initial clustering attention layer). These properties bring state-of-the-art UATs closer to practical machine learning, shifting the central open question from algorithmic implementability with noisy samples to whether stochastic gradient descent can achieve comparable guarantees.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。