让图像生成可变长度的表示，复杂图像用更多令牌，更高效准确。
Images are Worth Variable Length of Representations
- 根据图像复杂度动态生成不同数量的视觉令牌
- 平均令牌数减少，重建质量仍高，下游任务表现更好
- 支持查询引导，聚焦关键区域，适合多模态应用
现有视觉编码器将图像映射为固定长度的令牌序列，忽略了不同图像的信息量差异。例如，杂乱房间这类视觉复杂的图像本应比空白墙壁等简单图像承载更多信息，理应分配更多令牌。为此，我们提出 DOVE，一种动态视觉编码器，能为每张图像生成可变数量的视觉令牌（即连续表示向量），以实现图像重建。实验表明，DOVE 显著减少了平均令牌数量，同时保持高重建质量。在多个线性探测和下游多模态任务中，使用远少于现有自编码器基方法的令牌时，其表现更优，捕获到更具表达力的语义特征。我们进一步扩展了 DOVE，引入查询条件令牌化机制，通过引导模型关注与查询相关的区域，实现更高效、精准的语义提取。代码与检查点已公开于 https://dove-encoder.github.io/dove-encoder。
原文摘要 · Abstract (English)
Most existing vision encoders map images into a fixed-length sequence of tokens, overlooking the fact that different images contain varying amounts of information. For example, a visually complex image (e.g., a cluttered room) inherently carries more information and thus deserves more tokens than a simple image (e.g., a blank wall). To address this inefficiency, we propose DOVE, a dynamic vision encoder that produces a variable number of visual tokens (i.e., continuous representation vectors) to reconstruct each image. Our results show that DOVE significantly reduces the average number of tokens while maintaining high reconstruction quality. In several linear probing and downstream multimodal tasks, it outperforms existing autoencoder-based tokenization methods when using far fewer tokens, capturing more expressive semantic features compared to fixed-length encoding. We further extend DOVE with query-conditioned tokenization. By guiding the model to focus on query-relevant regions, it achieves more efficient and targeted semantic extraction. Our code and checkpoints are available at https://dove-encoder.github.io/dove-encoder.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。