提出简单正则化缓解向量量化中的代码坍缩问题。
Pushing Toward the Simplex Vertices: A Simple Remedy for Code Collapse in Smoothed Vector Quantization
- 通过最小化单纯形顶点与最近K个平滑量化结果的距离，统一优化代码利用率与逼近精度。
- 在图像自编码和语音表示学习任务中，代码使用更均衡，性能优于已有方法。
- 方法简洁有效，适合需要稳定向量量化的深度学习场景。
向量量化将连续向量空间离散化为一组代表向量（码本），广泛应用于现代机器学习。尽管高效，其非可微的量化步骤会阻断梯度反向传播。平滑向量量化通过将码本向量的硬分配替换为码本条目的加权组合（即单纯形向量与码本的矩阵乘积）来解决此问题。有效平滑需满足两点：（1）平滑量化器应接近一热向量，以保证紧致逼近；（2）所有码本条目应被充分利用，防止代码坍缩。现有方法通常分别处理这两个目标。本文提出一种简单直观的正则化策略，通过最小化每个单纯形顶点与其K个最近平滑量化结果之间的距离，同时实现上述两个目标。在代表性基准测试（包括离散图像自编码和对比式语音表征学习）上的实验表明，该方法显著提升了码本利用的可靠性，并优于先前方法。
原文摘要 · Abstract (English)
Vector quantization, which discretizes a continuous vector space into a finite set of representative vectors (a codebook), has been widely adopted in modern machine learning. Despite its effectiveness, vector quantization poses a fundamental challenge: the non-differentiable quantization step blocks gradient backpropagation. Smoothed vector quantization addresses this issue by relaxing the hard assignment of a codebook vector into a weighted combination of codebook entries, represented as the matrix product of a simplex vector and the codebook. Effective smoothing requires two properties: (1) smoothed quantizers should remain close to a onehot vector, ensuring tight approximation, and (2) all codebook entries should be utilized, preventing code collapse. Existing methods typically address these desiderata separately. By contrast, the present study introduces a simple and intuitive regularization that promotes both simultaneously by minimizing the distance between each simplex vertex and its $K$-nearest smoothed quantizers. Experiments on representative benchmarks, including discrete image autoencoding and contrastive speech representation learning, demonstrate that the proposed method achieves more reliable codebook utilization and improves performance compared to prior approaches.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。