用可学习的正交基函数实现输入空间连续1D正则化，提升模型泛化能力。
Deep Sturm--Liouville: From Sample-Based to 1D Regularization with Learnable Orthogonal Basis Functions
- 基于斯图姆-刘维尔定理，在输入空间构建场线并生成正交基函数。
- 在MNIST和CIFAR-10上实现与主流方法相当性能，样本效率显著提升。
- 适合关注模型泛化性与连续正则化机制的研究者。
尽管人工神经网络在各类任务中取得显著成功，但仍存在泛化能力有限的问题。我们假设其根源在于传统基于样本（零维）的正则化方式。为此，提出一种新型函数逼近器——深度斯图姆-刘维尔（DSL），通过将斯图姆-刘维尔定理（SLT）融入深度学习框架，实现输入空间沿场线的连续1维正则化。DSL定义穿越输入空间的场线，并在每条线上求解斯图姆-刘维尔问题，生成具有正交性的基函数，利用SLT的良好性质实现隐式正则化。这些基函数通过神经网络参数化后线性组合构成DSL逼近器，场线与基函数共同学习。我们证明了DSL形式可自然从秩一抛物型特征值问题中导出，通过隐式微分实现高效随机梯度下降训练。在包括高维图像数据集（如MNIST、CIFAR-10）在内的多变量数据集上，DSL表现优异，且具备更强的样本效率。
原文摘要 · Abstract (English)
Although Artificial Neural Networks (ANNs) have achieved remarkable success across various tasks, they still suffer from limited generalization. We hypothesize that this limitation arises from the traditional sample-based (0--dimensionnal) regularization used in ANNs. To overcome this, we introduce \textit{Deep Sturm--Liouville} (DSL), a novel function approximator that enables continuous 1D regularization along field lines in the input space by integrating the Sturm--Liouville Theorem (SLT) into the deep learning framework. DSL defines field lines traversing the input space, along which a Sturm--Liouville problem is solved to generate orthogonal basis functions, enforcing implicit regularization thanks to the desirable properties of SLT. These basis functions are linearly combined to construct the DSL approximator. Both the vector field and basis functions are parameterized by neural networks and learned jointly. We demonstrate that the DSL formulation naturally arises when solving a Rank-1 Parabolic Eigenvalue Problem. DSL is trained efficiently using stochastic gradient descent via implicit differentiation. DSL achieves competitive performance and demonstrate improved sample efficiency on diverse multivariate datasets including high-dimensional image datasets such as MNIST and CIFAR-10.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。