通过正交向量分解，实现高效模型压缩与微调，显著提升推理效率。
CLOVER: Cross-Layer Orthogonal Vectors Pruning and Fine-Tuning
- 将注意力层配对进行奇异值分解，构建低秩近似结构
- 70%的键值对剪枝后性能接近传统方法8%剪枝效果
- 可直接用于大模型微调，适配多类生成模型
Decoder-only模型通过缓存键值向量自回归生成令牌，但缓存增长导致推理受内存限制。为此，我们提出CLOVER（跨层正交向量），将注意力层成对视为低秩分解集合。对每个注意力头内的Q-K和V-O对应用奇异值分解（SVD）。所得奇异值可用于指导剪枝，或作为可训练参数实现正交向量的高效微调。剪枝或微调后，这些值重新融入模型，不增加参数量。我们在GPT-2 XL、DeepSeek-V2-Lite、Whisper-Large-v3、Stable Diffusion XL和LLaMA-3.2-11B-Vision等模型上验证了该方法。结果表明，CLOVER显著提升了剪枝效率：例如，在GPT-2 XL中剪除70%的Q-K对，其困惑度接近于传统方法仅剪除8%的效果。进一步微调奇异值后，实现全秩更新，在八个常识任务上对LLaMA-2 7B的表现优于LoRA、DoRA、HiRA和PiSSA，分别提升7.6%、5.5%、3.8%和0.7%。
原文摘要 · Abstract (English)
Decoder-only models generate tokens autoregressively by caching key/value vectors, but as the cache grows, inference becomes memory-bound. To address this issue, we introduce CLOVER (Cross-Layer Orthogonal Vectors), a novel approach that treats pairs of attention layers as a set of low-rank decompositions. CLOVER applies Singular Value Decomposition (SVD) to the \( Q \)-\( K \) and \( V \)-\( O \) pairs within each attention head. The resulting singular values can either guide pruning or serve as trainable parameters for efficient fine-tuning of all orthogonal vectors. After pruning or fine-tuning, these values are reintegrated into the model without increasing its parameter count. We apply CLOVER to various models, including GPT-2 XL, DeepSeek-V2-Lite, Whisper-Large-v3, Stable Diffusion XL, and LLaMA-3.2-11B-Vision. Our results demonstrate that CLOVER significantly improves pruning efficiency. For instance, the perplexity of pruning 70\% of the \( Q \)-\( K \) pairs in GPT-2 XL is similar to that of pruning just 8\% with vanilla methods. Fine-tuning the singular values further results in a full-rank update, outperforming state-of-the-art methods (LoRA, DoRA, HiRA, and PiSSA) by 7.6\%, 5.5\%, 3.8\%, and 0.7\%, respectively, on eight commonsense tasks for LLaMA-2 7B.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。