用专家混合动态调整码本，让向量压缩更快更准。
RQ-MoE: Residual Quantization via Mixture of Experts for Efficient Input-Dependent Vector Compression
- 两级专家混合+双流量化，实现输入自适应码本
- 重建与检索性能达顶尖水平，解码速度提升6到14倍
- 适合需要高速压缩的场景，如大模型推理加速
向量量化是压缩高维嵌入的核心技术，但现有多码本方法依赖静态码本，在异构数据几何下表达能力受限。尽管近期动态量化器QINCo能按输入调整码本并提升表达力，但其严格的顺序依赖导致解码瓶颈。本文提出残差量化-专家混合（RQ-MoE）框架，结合两层专家混合与双流量化，实现输入相关的码本动态构建，并将指令与量化解耦，支持并行解码。理论上，我们证明标准残差量化和QINCo可作为RQ-MoE的约束特例，并推导出专家维度设置准则。大量实验表明，RQ-MoE在重建与检索任务上达到或超过当前最优表现，且解码速度比以往方法快6至14倍。代码已开源：https://github.com/KDEGroup/RQ-MoE。
原文摘要 · Abstract (English)
Vector quantization is a fundamental tool for compressing high-dimensional embeddings, yet existing multi-codebook methods rely on static codebooks that limit expressiveness under heterogeneous data geometry. While recent dynamic quantizers like QINCo adapt codebooks to individual inputs and improve expressiveness, their strict sequential dependencies create decoding bottlenecks. We propose Residual Quantization via Mixture of Experts (RQ-MoE), a framework combining a two-level MoE with dual-stream quantization to enable input-dependent codebook adaptation for efficient vector quantization. RQ-MoE enables dynamic codebook construction and decouples instruction from quantization, facilitating parallel decoding. Theoretically, we show that standard Residual Quantization and QINCo can be recovered as constrained special cases of RQ-MoE, and derive a guideline for setting expert dimensionality in RQ-MoE. Extensive experiments show that RQ-MoE achieves state-of-the-art or on-par performance in reconstruction and retrieval, while providing 6x-14x faster decoding than prior vector quantization methods. The implementation is available at https://github.com/KDEGroup/RQ-MoE.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。