arXiv:2603.26756cs.CV2026-03被引 1

用可调注意力替代固定残差连接,让网络自适应学习特征。

GradAttn: Replacing Fixed Residual Connections with Task-Modulated Attention Pathways

  • 用自注意力动态调节梯度流动,实现任务感知的特征融合。
  • 在8个数据集上优于ResNet-18,FashionMNIST提升达11.07%。
  • 适合需要自适应特征学习的深度模型设计者。

深层卷积网络随深度增加面临梯度信号退化问题,限制了复杂架构中的有效特征学习。虽然残差网络通过固定残差连接缓解此问题,但这些固定通路无法根据输入复杂度调整,也难于选择性强调任务相关特征。本文提出GradAttn,将卷积神经网络中的固定残差连接替换为受注意力控制的梯度路径。通过在不同深度提取多尺度特征并使用自注意力调节,动态加权浅层纹理与深层语义表示。在八个多样化数据集(包括自然图像、医学影像和时尚识别)上评估三种GradAttn变体,结果表明其在五个数据集上超越ResNet-18,FashionMNIST最高提升11.07%,且网络规模相当。梯度流分析显示,由注意力引入的可控不稳定性常伴随更好的泛化能力,挑战了“完美稳定最优”的假设。此外,位置编码效果因数据集而异,通常网络层级已具备足够空间结构编码能力。研究证明注意力机制可作为可学习梯度调控的工具,为深层神经网络提供新范式。

原文摘要 · Abstract (English)

Deep ConvNets suffer from gradient signal degradation as network depth increases, limiting effective feature learning in complex architectures. ResNet addressed this through residual connections, but these fixed short circuits cannot adapt to varying input complexity or selectively emphasize task-relevant features across network hierarchies. This study introduces GradAttn, a variation of the residual approach in CNNs that replaces the fixed residual connections with attention-controlled gradient flow. By extracting multi-scale CNN features at different depths and regulating them through self-attention, GradAttn dynamically weights shallow texture features and deep semantic representations. For representational analysis, we evaluated three GradAttn variants across eight diverse datasets: from natural images and medical imaging to fashion recognition. The results demonstrate that GradAttn outperforms ResNet-18 on five of eight datasets, achieving up to +11.07% accuracy improvement on FashionMNIST while maintaining a comparable network size. Gradient flow analysis reveals that controlled instabilities, introduced by attention, often coincide with improved generalization, challenging the assumption that perfect stability is optimal. Furthermore, positional encoding's effectiveness turned out to be dataset-dependent, with CNN hierarchies frequently encoding sufficient spatial structure. These findings render attention mechanisms as enablers of learnable gradient control, offering a new way for adaptive representation learning in deep neural architectures.

残差连接注意力机制特征融合

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。