CoQuant通过联合分析权重与激活的误差，提升低比特大模型推理精度。
CoQuant: Joint Weight-Activation Subspace Projection for Mixed-Precision LLMs
- 同时考虑权重和激活的量化噪声，优化高精度子空间选择。
- 在Llama-3.2和Qwen2.5上，比现有方法降低困惑度并提升推理准确率。
- 适合追求低比特部署但又不牺牲性能的模型压缩研究者。
后训练量化（PTQ）已成为降低大语言模型（LLM）推理成本的重要技术。尽管近期混合精度方法通过保留关键高精度子空间提升了超低比特量化效果，但其子空间构建通常仅依赖激活统计信息，忽略了线性运算中输出扰动由权重与激活量化噪声共同驱动的本质。本文提出CoQuant，一种联合权重-激活子空间投影方法。通过理论建模预期输出误差，CoQuant推导出闭式加权主成分分析解，平衡激活与权重协方差，以选择最优高精度子空间。在Llama-3.2和Qwen2.5模型上的大量实验表明，CoQuant在WikiText困惑度和零样本常识推理准确率上持续优于强基准方法。结果证明，联合权重-激活子空间建模为低比特LLM量化提供了原理清晰且有效的方向。源代码见https://github.com/Zachary5895/CoQuant。
原文摘要 · Abstract (English)
Post-training quantization (PTQ) has become an important technique for reducing the inference cost of Large Language Models (LLMs). While recent mixed-precision methods improve ultra-low bit quantization by preserving critical subspaces in high precision, they typically construct these subspaces relying solely on activation statistics. This ignores the fundamental nature of linear operations, where the output perturbation is jointly driven by both activation and weight quantization noise. In this paper, we propose CoQuant, a joint weight-activation subspace projection method. By theoretically modeling the expected output error, CoQuant formulates a closed-form weighted PCA solution that balances activation and weight covariances to select the optimal high-precision subspace. Extensive experiments on Llama-3.2 and Qwen2.5 models show that CoQuant consistently outperforms strong PTQ baselines in both WikiText perplexity and zero-shot common-sense reasoning accuracy. These results demonstrate that joint weight-activation subspace modeling provides a principled and effective direction for low-bit LLM quantization. The source code is available at https://github.com/Zachary5895/CoQuant.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。