arXiv:2507.04779stat.MLcs.LG2025-07

提出o1Neuro模型,实现深度网络函数逼近与样本收敛保障

Constructive Universal Approximation and Sure Convergence for Multi-Layer Neural Networks

  • 基于稀疏指示激活神经元构建深度网络,支持函数逼近与交互建模
  • 在样本量10000时,对复杂回归任务预测性能优于XGBoost等基线模型
  • 理论保证收敛性,适用于高维交互数据建模,适合机器学习研究者

我们提出o1Neuro,一种基于稀疏指示激活神经元的新神经网络模型,具备两项关键统计性质:(1) 构造性通用逼近:在总体层面,深层o1Neuro可逼近任意可测函数(假设输入∈[0,1]^p且密度有界);浅层o1Neuro足以处理含两两交互项的加法模型,包括XOR和单变量项。结合已有工作表明单隐层非稀疏网络为通用逼近器,揭示了激活稀疏性与网络深度在逼近能力上的权衡。(2) 确定性收敛:在样本层面,经过足够多次更新后,o1Neuro优化可达到最优模型,概率趋近于1;并给出线性生成模型下所需更新次数有界的例子。实验上,将o1Neuro与XGBoost、随机森林及TabNet比较,在OpenML与UCI基准数据集(n=10000)以及合成数据集(100≤n≤20000)上展示出更优的预测性能。

原文摘要 · Abstract (English)

We propose o1Neuro, a new neural network model built on sparse indicator activation neurons, with two key statistical properties. (1) Constructive universal approximation: At the population level, a deep o1Neuro can approximate any measurable function of $\boldsymbol{X}$, while a shallow o1Neuro suffices for additive models with two-way interaction components, including XOR and univariate terms, assuming $\boldsymbol{X} \in [0,1]^p$ has bounded density. Combined with prior work showing that a single-hidden-layer non-sparse network is a universal approximator, this highlights a trade-off between activation sparsity and network depth in approximation capability. (2) Sure convergence: At the sample level, the optimization of o1Neuro reaches an optimal model with probability approaching one after sufficiently many update rounds, and we provide an example showing that the required number of updates is well bounded under linear data-generating models. Empirically, o1Neuro is compared with XGBoost, Random Forests, and TabNet for learning complex regression functions with interactions, demonstrating superior predictive performance on several benchmark datasets from OpenML and the UCI Machine Learning Repository with $n = 10000$, as well as on synthetic datasets with $100 \le n \le 20000$.

神经网络函数逼近模型收敛机器学习

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。