通过张量分解压缩注意力层，显著提升大模型推理能力且无需重新训练。
TensorLLM: Tensorising Multi-Head Attention for Enhanced Reasoning and Compression in LLMs
- 将多头注意力权重进行张量化与特克分解，实现结构化降噪与压缩。
- 注意力层压缩率达约250倍，推理性能在多个数据集上持续提升。
- 可无缝融合现有前馈网络降噪方法，适合追求高效推理的部署场景。
大型语言模型（LLMs）的推理能力可通过结构化去噪其权重得到提升，但现有技术主要针对Transformer块中的前馈网络（FFN），难以高效利用作为Transformer核心的多头注意力（MHA）模块。为此，我们提出一种新颖直观的框架，核心在于通过多头张量化与Tucker分解对MHA进行压缩，强制多个注意力头共享高维子空间，从而实现更高维度的结构化去噪与压缩。实验表明，该方法在多个基准数据集上，对编码器仅用和解码器仅用架构均能持续提升模型推理能力，同时使MHA权重压缩率达约250倍，且无需额外数据、训练或微调。此外，该方法可无缝结合现有基于FFN的去噪技术，进一步提升推理性能。
原文摘要 · Abstract (English)
The reasoning abilities of Large Language Models (LLMs) can be improved by structurally denoising their weights, yet existing techniques primarily focus on denoising the feed-forward network (FFN) of the transformer block, and can not efficiently utilise the Multi-head Attention (MHA) block, which is the core of transformer architectures. To address this issue, we propose a novel intuitive framework that, at its very core, performs MHA compression through a multi-head tensorisation process and the Tucker decomposition. This enables both higher-dimensional structured denoising and compression of the MHA weights, by enforcing a shared higher-dimensional subspace across the weights of the multiple attention heads. We demonstrate that this approach consistently enhances the reasoning capabilities of LLMs across multiple benchmark datasets, and for both encoder-only and decoder-only architectures, while achieving compression rates of up to $\sim 250$ times in the MHA weights, all without requiring any additional data, training, or fine-tuning. Furthermore, we show that the proposed method can be seamlessly combined with existing FFN-only-based denoising techniques to achieve further improvements in LLM reasoning performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。