用梯度信息增强残差连接,更好捕捉高频函数细节。
Gradient Residual Connections
- 引入梯度残差连接,补充传统恒等跳接。
- 在高频正弦回归任务中,误差降低42%以上。
- 适合图像超分辨率等高频重建任务。
现有研究指出函数梯度特性与函数逼近难度相关。受此启发,我们探究如何利用梯度信息提升神经网络对高频函数的逼近能力,并提出一种基于梯度的残差连接,作为残差网络中标准恒等跳接的补充。我们提供了简洁的理论直觉,说明梯度信息有助于区分输入,改善快速变化函数的逼近效果。在一个具有高频正弦真实值的合成回归任务中,传统残差连接难以捕捉高频模式;而我们的梯度残差显著提升逼近质量。随后,我们引入标准残差与梯度残差的凸组合,使网络可灵活控制梯度信息的依赖程度。通过消融实验验证设计合理性后,进一步在单图超分辨率任务上验证方法有效性,该任务中的潜在函数可能为高频。最后,在图像分类与分割等标准任务上,本方法性能与标准残差网络相当,表明其广泛适用性。
原文摘要 · Abstract (English)
Existing work has linked properties of a function's gradient to the difficulty of function approximation. Motivated by these insights, we study how gradient information can be leveraged to improve neural network's ability to approximate high-frequency functions, and we propose a gradient-based residual connection as a complement to the standard identity skip connection used in residual networks. We provide simple theoretical intuition for why gradient information can help distinguish inputs and improve the approximation of functions with rapidly varying behaviour. On a synthetic regression task with a high-frequency sinusoidal ground truth, we show that conventional residual connections struggle to capture high-frequency patterns. In contrast, our gradient residual substantially improves approximation quality. We then introduce a convex combination of the standard and gradient residuals, allowing the network to flexibly control how strongly it relies on gradient information. After validating the design choices of our proposed method through an ablation study, we further validate our approach's utility on the single-image super-resolution task, where the underlying function may be high-frequency. Finally, on standard tasks such as image classification and segmentation, our method achieves performance comparable to standard residual networks, suggesting its broad utility.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。