YOCO++通过残差连接提升压缩版大模型推理性能
YOCO++: Enhancing YOCO with KV Residual Connections for Efficient LLM Inference
- 在底层与中层间引入加权残差连接，增强键值缓存信息保留
- 50%缓存压缩率下性能超越标准Transformer，达当前最优
- 无需额外训练成本，适合部署资源受限的大模型
跨层键值（KV）压缩已被证明可有效降低大语言模型（LLMs）的推理内存开销。尽管此类方法能减少KV缓存占用，但通常会带来显著性能下降。本文旨在提升YOCO这一跨层KV压缩方法的性能，该方法将中间层的KVs共享给上半部分层。我们提出YOCO++，在每个下半层的KVs与底层之间引入加权残差连接。相较于YOCO，YOCO++在保持相同训练和推理效率的同时提升了模型容量。实验表明，在50%的KV缓存压缩率下，YOCO++在同类压缩方法中达到最先进性能，优于标准Transformer。
原文摘要 · Abstract (English)
Cross-layer key-value (KV) compression has been found to be effective in efficient inference of large language models (LLMs). Although they reduce the memory consumption of the KV cache, such methods usually introduce non-negligible performance degradation. In this work, we aim to enhance the performance of YOCO, a cross-layer KV compression method that shares the KVs of the middle layer with the top-half layers. We propose YOCO++, an enhanced YOCO that incorporates a weighted residual connection between the KVs of each bottom-half layer and the bottom layer. Compared to YOCO, YOCO++ increases model capacity while maintaining the same training and inference efficiency. Our experiments show that YOCO++ achieves state-of-the-art performance among the cross-layer KV compression methods at a 50% KV cache compression rate, outperforming the standard Transformer.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。