系统梳理多模态大模型离散化表示方法,解析向量量化技术如何提升效率与兼容性。
Discrete Tokenization for Multimodal LLMs: A Comprehensive Survey
- 基于向量量化构建离散令牌,适配语言模型处理流程
- 对比8种典型方法,揭示训练动态与集成挑战
- 适合研究多模态融合、模型压缩与高效生成的学者
大语言模型(LLMs)的快速发展催生了将连续多模态数据转化为适合语言处理的离散表示的需求。离散化通过向量量化(VQ)实现,兼具计算效率与对LLM架构的兼容性。尽管重要性日益凸显,现有研究缺乏对面向LLM系统的VQ技术的系统性综述。本文首次提出结构化分类体系,分析8种代表性VQ变体,涵盖经典与现代范式,考察其算法原理、训练动态及与LLM流水线的集成难题。进一步从非LLM场景、单模态LLM系统和多模态LLM系统三个维度梳理研究成果,揭示量化策略对对齐、推理与生成性能的影响。识别出代码本坍缩、梯度估计不稳、模态特异性编码约束等关键挑战,并探讨动态自适应量化、统一令牌框架与类脑码本学习等新兴方向。该综述连接传统向量量化与现代大模型应用,为构建高效通用的多模态系统提供基础参考。持续更新版本见:https://github.com/jindongli-Ai/LLM-Discrete-Tokenization-Survey。
原文摘要 · Abstract (English)
The rapid advancement of large language models (LLMs) has intensified the need for effective mechanisms to transform continuous multimodal data into discrete representations suitable for language-based processing. Discrete tokenization, with vector quantization (VQ) as a central approach, offers both computational efficiency and compatibility with LLM architectures. Despite its growing importance, there is a lack of a comprehensive survey that systematically examines VQ techniques in the context of LLM-based systems. This work fills this gap by presenting the first structured taxonomy and analysis of discrete tokenization methods designed for LLMs. We categorize 8 representative VQ variants that span classical and modern paradigms and analyze their algorithmic principles, training dynamics, and integration challenges with LLM pipelines. Beyond algorithm-level investigation, we discuss existing research in terms of classical applications without LLMs, LLM-based single-modality systems, and LLM-based multimodal systems, highlighting how quantization strategies influence alignment, reasoning, and generation performance. In addition, we identify key challenges including codebook collapse, unstable gradient estimation, and modality-specific encoding constraints. Finally, we discuss emerging research directions such as dynamic and task-adaptive quantization, unified tokenization frameworks, and biologically inspired codebook learning. This survey bridges the gap between traditional vector quantization and modern LLM applications, serving as a foundational reference for the development of efficient and generalizable multimodal systems. A continuously updated version is available at: https://github.com/jindongli-Ai/LLM-Discrete-Tokenization-Survey.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。