针对RWKV模型量化性能下降问题，提出自适应混合量化方法。
RWKVQuant: Quantizing the RWKV Family with Proxy Guided Hybrid of Scalar and Vector Quantization
- 设计粗粒度到细粒度代理，动态选择量化策略
- 3比特量化下精度损失低于1%，推理提速2.14倍
- 适合边缘设备部署的大型RWKV模型量化
RWKV是一种性能媲美Transformer的现代RNN架构，但在资源受限设备上部署仍面临挑战。后训练量化（PTQ）是降低模型大小和推理延迟的关键技术，已在Transformer模型中广泛应用。然而，将其应用于RWKV时性能显著下降。本文分析发现两大内在限制：(1) 非线性算子阻碍平滑与旋转量化中的参数融合，引入额外计算开销；(2) 大量均匀分布的权重给基于聚类的量化带来困难，导致精度下降。为此，我们提出RWKVQuant，一种专为RWKV设计的PTQ框架，包含两项新技术：(1) 一种粗粒度到细粒度的代理机制，通过评估权重均匀性和识别异常值，自适应选择不同量化方式；(2) 一种码本优化算法，提升RWKV中逐元素乘法的聚类量化性能。实验表明，RWKVQuant可将RWKV-6-14B量化至约3比特，精度损失小于1%，推理速度提升2.14倍。
原文摘要 · Abstract (English)
RWKV is a modern RNN architecture with comparable performance to Transformer, but still faces challenges when deployed to resource-constrained devices. Post Training Quantization (PTQ), which is a an essential technique to reduce model size and inference latency, has been widely used in Transformer models. However, it suffers significant degradation of performance when applied to RWKV. This paper investigates and identifies two key constraints inherent in the properties of RWKV: (1) Non-linear operators hinder the parameter-fusion of both smooth- and rotation-based quantization, introducing extra computation overhead. (2) The larger amount of uniformly distributed weights poses challenges for cluster-based quantization, leading to reduced accuracy. To this end, we propose RWKVQuant, a PTQ framework tailored for RWKV models, consisting of two novel techniques: (1) a coarse-to-fine proxy capable of adaptively selecting different quantization approaches by assessing the uniformity and identifying outliers in the weights, and (2) a codebook optimization algorithm that enhances the performance of cluster-based quantization methods for element-wise multiplication in RWKV. Experiments show that RWKVQuant can quantize RWKV-6-14B into about 3-bit with less than 1% accuracy loss and 2.14x speed up.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。