DA4通过关注微调参数方向，实现大模型权重量化后性能不降。
DAQ: Delta-Aware Quantization for Post-Training LLM Weight Compression
- 用参数增量方向一致性替代传统重建误差优化
- 在FP8量化下恢复标准方法丢失的风格化能力
- 无需数据即可压缩，适合部署阶段模型瘦身
我们提出一种无数据的后训练量化解法DA4，旨在保留后训练阶段学到的知识。标准量化的目标是最小化重构误差，但对基础模型不敏感，导致量化的噪声会不成比例地破坏编码后训练行为的小幅参数增量（ΔW），这一现象我们从量化的隐式正则化视角进行分析。DA4采用两种增量感知指标——符号保持率与余弦相似度——直接优化ΔW的方向保真度，仅需基础模型和后训练后的权重矩阵。在初步的FP8实验中，DA4恢复了标准量化下丢失的风格化能力，同时维持了整体性能。
原文摘要 · Abstract (English)
We introduce Delta-Aware Quantization (DAQ), a data-free post-training quantization framework that preserves the knowledge acquired during post-training. Standard quantization objectives minimize reconstruction error but are agnostic to the base model, allowing quantization noise to disproportionately corrupt the small-magnitude parameter deltas ($ΔW$) that encode post-training behavior -- an effect we analyze through the lens of quantization as implicit regularization. DAQ replaces reconstruction-based objectives with two delta-aware metrics -- Sign Preservation Rate and Cosine Similarity -- that directly optimize for directional fidelity of $ΔW$, requiring only the base and post-trained weight matrices. In a pilot FP8 study, DAQ recovers style-specific capabilities lost under standard quantization while maintaining general performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。