用二次变换提升神经网络非线性，小改动带来大性能提升
QuadEnhancer: Leveraging Quadratic Transformations to Enhance Deep Neural Networks
- 在每层引入特征间二次交互，增强模型表达能力
- 三类任务中均实现显著性能提升，参数与计算量几乎不变
- 适合想轻量改进现有模型的开发者和研究者
线性变换与非线性激活函数的结合构成了现代深度神经网络的基础，使其能够逼近高度复杂的函数。本文探索引入二次变换以进一步增强神经网络的非线性，旨在提升现有架构的性能。为降低参数与计算复杂度，提出一种轻量级二次增强模块，采用低秩、权值共享和稀疏化技术。对于固定架构，该方法在每层引入特征间的二次交互，仅增加可忽略的额外参数和前向计算开销。我们在图像分类、文本分类及大语言模型微调三个任务上进行了验证，结果表明该方法在所有任务中均展现出明显且显著的性能提升。
原文摘要 · Abstract (English)
The combination of linear transformations and non-linear activation functions forms the foundation of most modern deep neural networks, enabling them to approximate highly complex functions. This paper explores the introduction of quadratic transformations to further increase nonlinearity in neural networks, with the aim of enhancing the performance of existing architectures. To reduce parameter complexity and computational complexity, we propose a lightweight quadratic enhancer that uses low-rankness, weight sharing, and sparsification techniques. For a fixed architecture, the proposed approach introduces quadratic interactions between features at every layer, while only adding negligible amounts of additional model parameters and forward computations. We conduct a set of proof-of-concept experiments for the proposed method across three tasks: image classification, text classification, and fine-tuning large-language models. In all tasks, the proposed approach demonstrates clear and substantial performance gains.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。