揭示低比特量化中部分输入出错的原因及关键模块
Why Do Some Inputs Break Low-Bit LLM Quantization?
- 分析3-4比特量化方法，发现误差与残差流幅值相关
- 50组方法误差相关性达0.82，验证结果可复现
- 晚期层残差激活和MLP门控对性能维持最关键
低比特权重量化显著降低大语言模型（LLM）的内存占用，但对某些输入的影响尤为严重。我们在7B-70B规模的LLM上分析多种3-4比特方法，发现50组方法在FineWeb数据上的量化误差高度相关（平均0.82）。此外，全精度模型的残差流幅值可预示未来量化误差。我们提出假设：残差流幅值与层间误差放大和累积有关。通过模型定位、提前退出和激活修补技术，我们发现高误差样本依赖于晚期层的精确残差激活，且MLP门控输出对维持困惑度至关重要。本研究揭示了为何某些输入导致大量化误差，以及哪些模型组件对性能保持最为关键。
原文摘要 · Abstract (English)
Low-bit weight-only quantization significantly reduces the memory footprint of large language models (LLMs), but disproportionately affects certain examples. We analyze diverse 3-4 bit methods on LLMs ranging from 7B-70B in size and find that the quantization errors of 50 pairs of methods are strongly correlated (avg. 0.82) on FineWeb examples. Moreover, the residual stream magnitudes of full-precision models are indicative of future quantization errors. We further establish a hypothesis that relates the residual stream magnitudes to error amplification and accumulation over layers. Using LLM localization techniques, early exiting, and activation patching, we show that examples with large errors rely on precise residual activations in the late layers, and that the outputs of MLP gates play a crucial role in maintaining the perplexity. Our work reveals why certain examples result in large quantization errors and which model components are most critical for performance preservation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。