让视觉语言模型在压缩图像中智能选择性展开关键内容,提升文本理解效率。
LensVLM: Selective Context Expansion for Compressed Visual Representation of Text

- 通过学习工具动态选择性展开压缩图像中的关键区域,避免整体解码开销。
- 在4.3倍有效压缩下保持接近全量文本的准确率,10.1倍压缩时仍优于基线。
- 适用于文档、代码等多模态任务,尤其适合布局信息重要的场景。
视觉语言模型(VLMs)可将文本作为图像处理,无需将其分词为长序列。由于视觉编码器将固定尺寸图像映射为固定数量的视觉标记,不同的渲染分辨率提供了细粒度的压缩控制。然而,随着压缩增加,准确率迅速下降:字符缩小至视觉编码器的有效分辨率以下,难以区分。为此,我们提出LensVLM,一种推理框架与后训练方法,使VLM能够扫描压缩图像,并通过学习的工具仅将相关区域展开至未压缩形式。基于Qwen3.5-9B-Base,LensVLM在4.3倍有效压缩下保持与全文本上限相当的准确性,在七个文本问答基准上,于10.1倍有效压缩下超越基于检索、文本及视觉压缩的基线。LensVLM还泛化到多模态文档与代码理解任务,且压缩越大,相对于基线的性能增益越显著。分析表明:训练使视觉压缩对渲染选择更具鲁棒性;随着压缩加剧,模型更依赖展开内容而非不可靠的视觉阅读。分析还提供实用工具选择建议:渲染文本宜用文本展开,而原生文档则更适合高分辨率图像展开以保留布局信息。
原文摘要 · Abstract (English)
Vision Language Models (VLMs) offer the exciting possibility of processing text as rendered images, bypassing the need for tokenizing the text into long token sequences. Since VLM image encoders map fixed-size images to a fixed number of visual tokens, varying rendering resolution provides a fine-grained compression knob. However, accuracy deteriorates quickly as compression increases: characters shrink below the vision encoder's effective resolution, making them indistinguishable. To address this, we propose LensVLM, an inference framework and post-training recipe that enables VLMs to scan compressed images, then selectively expand only the relevant images to their uncompressed form via learned tools. Building on Qwen3.5-9B-Base, LensVLM maintains accuracy comparable to the full-text upper bound at 4.3x effective compression and outperforms retrieval-based, text- and visual-compression baselines up to 10.1x effective compression across seven text QA benchmarks. LensVLM also generalizes to multimodal document and code understanding tasks, with the accuracy gain over baselines growing as compression increases. Our analysis validates this approach: training makes visual compression robust to rendering choices, and as compression grows the model increasingly relies on expanded content rather than unreliable visual reading. The analysis also yields practical tool-choice guidance: text expansion is preferable for rendered text, while high-resolution image expansion suits native documents whose layout cues carry task-relevant information.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。