用更少参数提升模型性能，可替代传统残差连接。
LAuReL: Learned Augmented Residual Layer
- 提出新型残差层LAuReL，动态增强信息流路径。
- 在图像和语言模型上显著提效，参数增量不足0.1%。
- 适合追求轻量化高性能模型的开发者使用。
高效深度学习的核心之一是架构改进，如残差/跳跃连接，显著提升了模型收敛速度与质量。自该技术出现以来，残差连接已广泛应用于卷积神经网络及基于Transformer的架构，成为大模型的基石。本文提出一种新方法——学习增强残差层（LAuReL），作为经典残差连接的通用替代方案，在不牺牲性能的前提下优化模型质量与参数规模。实验表明，使用LAuReL可在保持极低参数开销的同时提升视觉与语言模型表现。例如在ResNet-50与ImageNet 1K任务中，仅增加0.003%参数，即可获得相当于加一层的60%性能提升；当预训练1B与4B参数的LLM时，下游任务性能提升2.54%至20.05%，额外参数仅增加0.012%与0.1%。
原文摘要 · Abstract (English)
One of the core pillars of efficient deep learning methods is architectural improvements such as the residual/skip connection, which has led to significantly better model convergence and quality. Since then the residual connection has become ubiquitous in not just convolutional neural networks but also transformer-based architectures, the backbone of LLMs. In this paper we introduce Learned Augmented Residual Layer (LAuReL) -- a novel generalization of the canonical residual connection -- with the goal to be an in-situ replacement of the latter while outperforming on both model quality and footprint metrics. Our experiments show that using LAuReL can help boost performance for both vision and language models. For example, on the ResNet-50, ImageNet 1K task, it achieves 60% of the gains from adding an extra layer, while only adding 0.003% more parameters, and matches it while adding 2.6 times fewer parameters. Similarly, when pre-training 1B and 4B parameter LLMs, LAuReL improves performance on a variety of challenging downstream evaluation tasks by 2.54% to 20.05%, while adding only 0.012% and 0.1% additional parameters, respectively.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。