通过引入值残差连接，提升深层Transformer的信息传递效率。
Value Residual Learning
- 在隐藏状态残差外增加值残差，改善信息流动
- 参数减少16.11%，训练数据少20.3%，性能相当
- 适合追求高效推理与低显存的模型部署场景
尽管Transformer在多个领域取得显著成功，但深层网络中的信息传播效果仍是关键挑战。标准隐藏状态残差难以在深层保持初始标记级信息。本文提出ResFormer，通过在隐藏状态残差基础上引入值残差连接来增强信息流；其变体SVFormer则让所有层共享第一层的值嵌入。实证结果表明，ResFormer在验证损失相当的情况下，模型参数减少16.11%，训练数据需求降低20.3%，同时保持相似的内存使用和计算开销。此外，SVFormer将键值缓存大小几乎减半，仅带来小幅性能损失，且可与其他KV高效方法结合，进一步压缩缓存，性能受序列长度和累积学习率影响。
原文摘要 · Abstract (English)
While Transformer models have achieved remarkable success in various domains, the effectiveness of information propagation through deep networks remains a critical challenge. Standard hidden state residuals often fail to adequately preserve initial token-level information in deeper layers. This paper introduces ResFormer, a novel architecture that enhances information flow by incorporating value residual connections in addition to hidden state residuals. And a variant is SVFormer, where all layers share the first layer's value embedding. Comprehensive empirical evidence demonstrates ResFormer achieves equivalent validation loss with 16.11\% fewer model parameters and 20.3\% less training data compared to Transformer, while maintaining similar memory usage and computational cost. Besides, SVFormer reduces KV cache size by nearly half with only a small performance penalty and can be integrated with other KV-efficient methods, yielding further reductions in KV cache, with performance influenced by sequence length and cumulative learning rate.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。