发现量化导致代码本萎缩，提出简单方法保持生成多样性
Early Quantization Shrinks Codebook: A Simple Fix for Diversity-Preserving Tokenization
- 在生成模型中发现量化会引发代码本与嵌入层双重坍塌
- 随机初始化和编码器容量不足是坍塌主因，影响生成质量
- 通过改进初始化和结构设计，有效防止多样性丢失
向量量化是将连续表示离散化为一组离散向量的技术，在大语言模型、扩散模型等生成模型中广泛应用。尽管如此，其在生成模型中的特性与行为仍缺乏深入研究。本文系统研究了向量量化中的坍塌问题，包括代码本令牌与连续潜在嵌入的坍塌。基于合成与真实数据集分析，揭示了随机初始化及编码器容量受限是导致两类坍塌的关键因素。据此提出针对性缓解方案。据我们所知，这是首个全面探讨向量量化中表示坍塌问题的研究。
原文摘要 · Abstract (English)
Vector quantization is a technique in machine learning that discretizes continuous representations into a set of discrete vectors. It is widely employed in tokenizing data representations for large language models, diffusion models, and other generative models. Despite its prevalence, the characteristics and behaviors of vector quantization in generative models remain largely underexplored. In this study, we systematically investigate the issue of collapses in vector quantization, where collapsed representations are observed across discrete codebook tokens and continuous latent embeddings. By leveraging both synthetic and real datasets, we identify the severity of each type of collapses and triggering conditions. Our analysis reveals that random initialization and limited encoder capacity result in tokens collapse and embeddings collapse. Building on these findings, we propose potential solutions aimed at mitigating each collapse. To the best of our knowledge, this is the first comprehensive study examining representation collapsing problems in vector quantization.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。