通过层间权重共享与低秩差异矩阵，压缩大模型内存占用。
DeltaLLM: Compress LLMs with Low-Rank Deltas between Shared Weights
- 采用层间共享权重+低秩差异矩阵结构，实现高效模型压缩。
- 参数减少12%仍保持90%原始性能，2.9B模型比3.3B小400M却更优。
- 无需微调即超越其他压缩方法，适合存储受限场景。
我们提出DeltaLLM，一种新的后训练压缩技术，用于降低大语言模型的内存占用。该方法在相邻Transformer块之间引入权重共享，并添加低秩差异矩阵。训练时采用渐进模块替换策略，仅需约3000万至4000万词元即可达到与从头训练相当的性能。我们发布了压缩后的DeltaLLAMA和DeltaPHI模型，参数量减少12%，在通用知识和推理基准上保留了基线Llama和Phi模型90%的性能。该方法在相同删减参数量下优于JointDrop、LaCo、ShortGPT和SliceGPT。例如，参数减少24%的DeltaPhi 2.9B，在零样本平均准确率上媲美经微调的SlicedPhi 3.3B（仅减少12%），且模型规模小约4亿参数，未经过微调。本工作为存储受限下的大模型架构设计与压缩提供了新思路。
原文摘要 · Abstract (English)
We introduce DeltaLLM, a new post-training compression technique to reduce the memory footprint of LLMs. We propose an alternative way of structuring LLMs with weight sharing between layers in subsequent Transformer blocks, along with additional low-rank difference matrices between them. For training, we adopt the progressing module replacement method and show that the lightweight training of the low-rank modules with approximately 30M-40M tokens is sufficient to achieve performance on par with LLMs of comparable sizes trained from scratch. We release the resultant models, DeltaLLAMA and DeltaPHI, with a 12% parameter reduction, retaining 90% of the performance of the base Llama and Phi models on common knowledge and reasoning benchmarks. Our method also outperforms compression techniques JointDrop, LaCo, ShortGPT and SliceGPT with the same number of parameters removed. For example, DeltaPhi 2.9B with a 24% reduction achieves similar average zero-shot accuracies as recovery fine-tuned SlicedPhi 3.3B with a 12% reduction, despite being approximately 400M parameters smaller with no fine-tuning applied. This work provides new insights into LLM architecture design and compression methods when storage space is critical.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。