提出三种向量量化压缩方法，提升模型训练稳定性与压缩效率。
Efficient VQ-QAT and Mixed Vector/Linear quantized Neural Networks
- 用余弦相似度分配向量，避免代码本坍缩，支持端到端训练。
- 结合顶1采样与直通估计，无需加权重建，简化量化流程。
- 基于可微神经架构搜索自适应选择每层量化配置，优化压缩效果。
本文研究并测试了三种基于向量量化（VQ）的模型权重压缩技术。为缓解代码本坍缩问题并支持端到端训练，采用基于余弦相似度的向量分配策略。借鉴可微K均值（DKM）中的注意力机制思想，进一步改进该方法：使用余弦相似度进行分配，结合顶1采样与直通估计，从而无需加权平均重建。最后，探索利用可微神经架构搜索（NAS）自适应选择各层量化配置，以优化压缩过程。尽管本方法在所有量化级别上未始终优于现有方法，但为VQ-based模型压缩的设计权衡与行为提供了有益洞见。
原文摘要 · Abstract (English)
In this work, we developed and tested 3 techniques for vector quantization (VQ) based model weight compression. To mitigate codebook collapse and enable end-to-end training, we adopted cosine similarity-based assignment. Building on ideas from attention-based formulations in Differentiable K-Means (DKM), we further improved this approach by using cosine similarity for assignment combined with top-1 sampling and a straight-through estimator, thereby eliminating the need for weighted-average reconstruction. Finally, we investigated the use of differentiable neural architecture search (NAS) to adaptively select layer-wise quantization configurations, further optimizing the compression process. Although our method does not consistently outperform existing approaches across all quantization levels, it provides useful insights into the design trade-offs and behaviors of VQ-based model compression methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。