让图像像文字一样离散化,还能保持细节与语义
ViQ: Text-Aligned Visual Quantized Representations at Any Resolution

- 分两阶段训练:先对齐文本增强语义,再逐步压缩特征空间
- 在任意分辨率下保持低级细节重建精度,性能媲美连续视觉模型
- 适合追求高效多模态训练的开发者,尤其适合大语言模型融合
统一文本与视觉的表示是自然追求,可简化多模态建模并提升训练效率。然而,将图像以离散信号形式表示会带来严重信息损失。现有方法难以平衡低级细节与高级语义:重建导向的表示缺乏语义,语义强的特征则严重丢失细节。本文提出ViQ,一种支持原生分辨率输入的视觉离散表示框架,可在保持语义与细节之间取得平衡。方法分为两个阶段:文本对齐预训练和特征离散化。通过预训练增强视觉编码器的语义监督,并使其能处理原生分辨率输入;在离散化阶段,提出近端表示学习策略以逐步压缩特征空间,结合位置感知的头级量化机制,实现任意分辨率灵活处理。大量多模态任务实验表明,ViQ在性能上可媲美使用连续高维视觉特征的前沿模型,同时保持高精度低级重建。此外,使用视觉离散表示进行多模态训练显著提升效率,相比不同基础大语言模型和训练方案,提速可达20%-70%。
原文摘要 · Abstract (English)
A unified representation for text and vision is a natural pursuit, as it enables simpler multimodal modeling and more efficient training. However, representing images as discrete signals in the same way as text inevitably introduces severe information loss. Existing work struggles to balance low-level details and high-level semantics in discrete representations: reconstruction-oriented representations often lack semantic information, whereas semantically stronger features typically suffer from severe loss of detail. We present ViQ, a Visual Quantized Representations framework, which is designed to balance semantics and details in discrete representations while supporting inputs at native resolutions, thereby enabling it to serve as a unified and general discrete representation for arbitrary visual inputs. Our approach structures quantization learning into two stages: text-aligned pre-training and feature discretization. With text-aligned pre-training, we enhance the visual encoder semantic-rich supervision from the pretrained language model and enable it to process native-resolution visual inputs. During discretization, we propose a proximal representation learning strategy to progressively compact the feature space, along with a position-aware head-wise quantization mechanism that enables flexible processing of arbitrary resolutions. Extensive experiments on multimodal tasks demonstrate that ViQ achieves competitive performance compared to state-of-the-art multimodal vision encoders with continuous and high-dimensional visual features, while maintaining high precision in low-level reconstruction. We also show that multimodal training with visual quantized representations largely improves efficiency, yielding up to 20\%-70\% acceleration with different base LLMs and training recipes.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。