提出ARCHead压缩LLM输出头，存储减3.9倍且性能几乎不变。
ARCHead: Activation-Metric Residual Correction for Large Language Model Output Heads
- 用低秩核心+分组INT4残差+激活度量修正，替代原BF16输出头
- 在Qwen3-8B上仅用25.6%存储量，相对困惑度达1.007（原为1.0）
- 兼容现有量化工具，替换后延迟变化<2%，适合部署优化
权重量化可大幅降低大语言模型变换器模块的存储开销，但实际后端常将最终的语言建模头（LM-head）保留为BF16或FP16。直接量化该投影会显著扰动词汇概率分布。本文提出ARCHead，一种打包的LM-head压缩方案，结合量化低秩核心、分组INT4残差及基于激活度量拟合的低秩修正。ARCHead不存储密集的BF16头，使持久化头存储减少3.7-3.9倍。在Qwen3-8B-Base上，其仅需25.6%的BF16头存储，相对困惑度达1.007；而存储匹配的朴素INT4方案困惑度为1.14-1.16。用其替换AWQ或bitsandbytes留下的BF16头，交叉熵增加仅0.006-0.007，吞吐变化低于2%。因此，ARCHead可与块级量化器协同，压缩它们无法处理的大型输出投影。代码已公开于https://github.com/suayptalha/archead。
原文摘要 · Abstract (English)
Weight-only quantization substantially reduces the storage of large language model (LLM) transformer blocks, but practical backends often retain the final language-modeling head (LM-head) in BF16 or FP16. Quantizing this projection naively can strongly perturb the vocabulary-logit distribution. We present ARCHead, a packed LM-head compressor that combines a quantized low-rank core, group-wise INT4 residuals, and a low-rank correction fitted in an activation-derived metric. ARCHead stores no dense BF16 head and reduces persistent LM-head storage by 3.7-3.9x. On Qwen3-8B-Base, it uses 25.6% of BF16 head storage while attaining 1.007 relative perplexity; storage-matched naive INT4 yields 1.14-1.16. Replacing the BF16 head left by AWQ or bitsandbytes adds only 0.006-0.007 cross-entropy, with less than 2% throughput change in our measurements. ARCHead therefore complements block quantizers by compressing the large output projection they can leave untouched. Code is available at https://github.com/suayptalha/archead.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。