Kelix实现全离散化多模态自回归建模，让视觉理解能力追平连续特征模型。
Kelix Technical Report
- 采用完全离散的视觉标记化，统一文本与视觉的序列建模
- 在多个基准上达到与连续特征模型相当的视觉理解性能
- 适合追求统一生成与理解框架的研究者或工业应用
自回归大语言模型通过将多样任务表达为自然语言离散标记序列，并以预测下一个标记的方式训练，在自监督下统一了理解和生成。将这一范式扩展到多模态数据需要跨模态的共享离散表示。然而，大多数视觉-语言模型仍依赖混合接口：离散文本标记搭配连续视觉变压器（ViT）特征。由于监督主要由文本驱动，这些模型往往偏向理解而无法充分利用非文本数据的大规模自监督学习。近期研究探索了离散视觉标记化以实现完全自回归的多模态建模，展现出向统一理解和生成迈进的潜力。但现有离散视觉标记常因代码容量有限导致信息丢失，使得理解能力明显弱于连续特征模型。我们提出Kelix，一个完全离散的自回归统一模型，弥合了离散与连续视觉表示之间的理解差距。
原文摘要 · Abstract (English)
Autoregressive large language models (LLMs) scale well by expressing diverse tasks as sequences of discrete natural-language tokens and training with next-token prediction, which unifies comprehension and generation under self-supervision. Extending this paradigm to multimodal data requires a shared, discrete representation across modalities. However, most vision-language models (VLMs) still rely on a hybrid interface: discrete text tokens paired with continuous Vision Transformer (ViT) features. Because supervision is largely text-driven, these models are often biased toward understanding and cannot fully leverage large-scale self-supervised learning on non-text data. Recent work has explored discrete visual tokenization to enable fully autoregressive multimodal modeling, showing promising progress toward unified understanding and generation. Yet existing discrete vision tokens frequently lose information due to limited code capacity, resulting in noticeably weaker understanding than continuous-feature VLMs. We present Kelix, a fully discrete autoregressive unified model that closes the understanding gap between discrete and continuous visual representations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。