LegoLM通过结构化权重共享，在不降精度前提下实现大模型高效压缩。
LegoLM: Structured Weight Sharing for Large Language Models
- 分块压缩中引入分层编码与异常值保护机制
- 在Mistral-7B上达5.12倍压缩率，困惑度仅增0.03%
- 适合需要极致压缩且保持精度的部署场景
我们提出LegoLM，一种基于对全局权重共享失败原因系统研究的结构化权重共享压缩框架。识别出两种失效模式：分布失配（当向量块维度d≤2时，异质缩放导致误差随d线性增长，无法通过增大K缓解）与异常值主导（标量块中约1/K权重超出聚类边界，跨层累积误差引发灾难性性能下降）。LegoLM通过三项无数据改进解决：1）标量块编码消除d线性失配；2）百分位选择性替换保留异常权重；3）首尾层边界保护。在GPT-2 small（124M）和Mistral-7B上，LegoLM在Mistral-7B上实现4.41倍压缩，困惑度仅+0.03%（优于PTQ-8bit），在K=64、p=99%时达5.12倍压缩，准确率与噪声相当，超越PTQ-8bit压缩比。发现异常值问题随模型规模加剧：全量替换在GPT-2 small上仅+23%，但在Mistral-7B上高达+1,134,279%；而选择性替换（p=99%）使两者均控制在+15%以内。消融实验确认选择性替换为关键机制：加入逐层K均值后即可接近LegoLM效果（差异<0.02%）。
原文摘要 · Abstract (English)
We present \LegoLM{}, a structured weight-sharing compression framework for large language models grounded in a systematic study of why global weight sharing fails and how to fix it. We identify two distinct failure modes. Distributional mismatch: for vector blocks of dimension d <= 2, transformer layers with heterogeneous weight scales impose a scale-mismatch penalty that grows linearly with d and cannot be resolved by increasing K, producing perplexity in the millions.Outlier dominance: for scalar blocks, a fraction ~1/K of weights lies beyond the outermost Lloyd-Max decision threshold and cannot be represented by any centroid; their misrepresentation accumulates across layers, causing catastrophic quality loss. \LegoLM{} resolves both failure modes via three data-free adaptations: 1 scalar-block encoding to eliminate the $d$-linear mismatch component, 2 percentile-selective replacement that identifies and preserves outlier weights verbatim, and 3 boundary-layer protection for the first and last transformer blocks. Across GPT-2 small (124M) and Mistral-7B, \LegoLM{} achieves +0.03% PPL degradation at 4.41X compression on Mistral-7B - outperforming PTQ-8bit in both quality and compression ratio - and -0.02% at 2.67X. Downstream evaluation on LAMBADA and HellaSwag confirms that \LegoLM{} at K=64, p=99% preserves accuracy within noise at 5.12 X compression, exceeding PTQ-8bit's compression ratio while matching its accuracy. We further discover that outlier dominance grows with model scale: full replacement at K=128 degrades GPT-2 small by only +23% but catastrophically degrades Mistral-7B by +1,134,279%, while selective replacement at p=99% rescues both models to under +15%. A controlled ablation confirms that selective replacement is the dominant mechanism: adding it to per-layer K-means also yields near-lossless quality, matching \LegoLM{} within 0.02%.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。