一个参数的删减能让大模型文本生成能力崩溃，发现关键参数与激活异常。
The Super Weight in Large Language Models
- 通过单次前向传播定位极少数关键参数（超权），可精准识别破坏性影响点。
- 删除任一超权参数使困惑度升高1000倍，零样本准确率降至随机猜测水平。
- 适用于模型压缩、量化优化研究，尤其适合关注模型鲁棒性与高效部署者。
近期研究发现，大型语言模型（LLM）中极少数参数异常值对模型质量具有不成比例的重要影响。由于LLM包含数十亿参数，这类占比仅0.01%的参数仍可达数十万级别。本文发现更惊人现象：仅删除一个参数即可导致模型文本生成能力完全丧失——困惑度上升三个数量级，零样本准确率降至随机猜测水平。为此提出一种无需数据的超权识别方法，仅需一次前向传播即可定位这些关键参数。我们进一步发现，超权会引发罕见且巨大的激活异常（称作超激活）。在高精度保留超激活的情况下，简单的四舍五入量化即可达到顶尖方法性能。对于权重量化，若保留超权并裁剪其他权重异常值，四舍五入量化可支持远大于以往的块大小。为推动后续研究，我们公开了常见开源大模型的超权坐标索引。
原文摘要 · Abstract (English)
Recent works have shown a surprising result: a small fraction of Large Language Model (LLM) parameter outliers are disproportionately important to the quality of the model. LLMs contain billions of parameters, so these small fractions, such as 0.01%, translate to hundreds of thousands of parameters. In this work, we present an even more surprising finding: Pruning as few as a single parameter can destroy an LLM's ability to generate text -- increasing perplexity by 3 orders of magnitude and reducing zero-shot accuracy to guessing. We propose a data-free method for identifying such parameters, termed super weights, using a single forward pass through the model. We additionally find that these super weights induce correspondingly rare and large activation outliers, termed super activations. When preserved with high precision, super activations can improve simple round-to-nearest quantization to become competitive with state-of-the-art methods. For weight quantization, we similarly find that by preserving the super weight and clipping other weight outliers, round-to-nearest quantization can scale to much larger block sizes than previously considered. To facilitate further research into super weights, we provide an index of super weight coordinates for common, openly available LLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。