让图像编码器根据文本查询生成视觉表示，提升多模态任务性能。
Text-Guided Semantic Image Encoder
- 图像编码器在文本引导下生成语义相关表示，实现动态感知。
- 在9个图文任务上分别提升1.5和1.3点，最高达6点增益。
- 仅需一半图像块，推理效率显著提高，适合实际部署。
图像编码器是视觉-语言模型（VLM）的基础组件，通常在与语言模型对齐前独立预训练。这种标准范式导致编码器处理图像时忽略下游任务或文本查询的具体需求。为解决该问题，我们提出文本引导的语义图像编码器（TIE），使图像表示生成过程受输入文本查询条件控制。配备TIE的VLM在1B和3B规模下，于九个图文基准测试中平均分别提升1.5和1.3点，部分任务如DocVQA和InfoVQA最高提升达6点。此外，基于TIE的VLM仅使用一半图像块（tokens），显著提升推理效率。TIE在通用查询下也表现出良好泛化能力，表明文本条件训练有效优化了编码器对关键视觉特征的捕捉。定性分析显示，TIE始终关注与查询相关的区域，增强可解释性与查询定位能力。
原文摘要 · Abstract (English)
Image encoders, a fundamental component of vision-language models (VLMs), are typically pretrained independently before being aligned with a language model. This standard paradigm results in encoders that process images agnostically, without regard to the specific downstream task or text query. To address this limitation, we propose the Text-Guided Semantic Image Encoder (TIE), which generates image representations conditioned on the input text query. VLMs equipped with TIE outperform their conventional counterparts by +1.5 and +1.3 points on average across nine image-to-text benchmarks at the 1B and 3B scales, respectively, with gains reaching up to 6 points on tasks such as DocVQA and InfoVQA. Moreover, TIE-based VLMs attain superior performance while utilizing only half as many image tiles (tokens), resulting in notably improved inference efficiency. TIE also generalizes well with generic queries, indicating that text-conditioned training effectively optimizes the encoder to capture key visual features. Qualitative analysis confirms that TIE consistently attends to query-relevant regions, enhancing both interpretability and query-specific grounding.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。