DReS通过双重建平滑提升模型泛化能力,无需额外参数且适配多种学习场景。
DReS: Dual Reconstruction Smoothing for Functional Regularization
- 用共享参数的样条辅助分支实现非参数平滑正则化
- 在自监督、生成和监督任务中均提升表示学习与生成质量
- 理论证明其为高阶光滑性隐式正则器,适合无监督学习
平滑性是机器学习中的关键归纳偏置,与泛化密切相关。现有平滑性正则方法通常依赖显式的梯度正则化(计算与内存开销大)或数据混合策略(不适用于无监督与自监督场景)。本文提出非参数正则框架DReS(Dual Reconstruction Smoothing),通过共享参数的样条辅助分支实现平滑性诱导,不引入额外可训练参数,可应用于任意子模块,适用于无监督、自监督及监督学习。理论上,目标函数与其DReS近似之间的误差由函数的高阶光滑性量度控制,表明该方法为隐式高阶光滑性正则器。实验显示,DReS在多个自监督方法中提升表示学习效果,在生成建模中改善生成质量,并在监督学习中达到优于基线的性能。
原文摘要 · Abstract (English)
Smoothness is a key inductive bias in machine learning and is closely related to generalization. Existing smoothness-inducing methods typically rely either on explicit gradient regularization, which often incurs substantial computational and memory overhead, or on data-mixing strategies, which are less naturally applicable to unsupervised and self-supervised settings. In this work, we propose $\textit{Dual Reconstruction Smoothing}$ (DReS), a nonparametric regularization framework that induces smoothness through a spline-based auxiliary branch with shared model parameters. The method introduces no additional trainable parameters and can be applied to arbitrary submodules, making it suitable for unsupervised, self-supervised, and supervised regimes. We show theoretically that the discrepancy between the target function and its DReS approximation is controlled by higher-order smoothness quantities of the function, establishing the method as an implicit higher-order smoothness regularizer. Empirically, DReS improves representation learning across several self-supervised methods, improves generation quality in generative modeling, and achieves strong performance relative to competitive baselines in supervised learning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。