让视觉语言模型更专注关键文字,自动放大重要段落提升理解力
Magnifying What Matters: Attention-Guided Adaptive Rendering for Visual Text Comprehension

- 用模型自身注意力识别关键文字区域,动态放大重渲染
- 在9个基准上提升效果,多页记忆问答任务增益显著
- 无需训练、适配多种模型,对图像模糊或文字错误有鲁棒性
视觉文本理解(VTC)将文本转为图像供视觉语言模型(VLM)阅读,突破大模型上下文窗口限制,适用于长文档OCR和多页记忆问答。现有方法将渲染与布局视为固定预处理,缺乏对VLM内部处理机制的理解。通过在VTC问答任务上的实证研究,我们发现VLM存在‘定位但不使用’现象:注意力集中在中后期层,却与答案正确性脱钩;仅放大这些定位区域即可恢复大量失败案例。基于此,我们提出AGAR(注意力引导自适应渲染),一种无需训练、模型无关的方法:利用VLM中后期注意力识别前K个重要视觉块,映射回对应词段,重新渲染时放大这些段落,再进行推理。在九个VTC基准(短文本、长上下文、多页记忆问答)和四种VLM骨干网络上实验表明,AGAR(i)作为即插即用模块持续提升现成VLM性能,(ii)可与VLM后训练结合进一步增益,(iii)在视觉与文本输入退化下仍保持鲁棒。
原文摘要 · Abstract (English)
Visual Text Comprehension (VTC) renders text into images for a vision-language model (VLM) to read, sidestepping LLM context-window limits and powering applications from long-page OCR to multi-page memory QA. Yet existing VTC pipelines treat rendering and layout as a fixed, content-agnostic preprocessing step and offer little mechanistic understanding of how VLMs internally process visualized text. Through a focused empirical study on VTC QA tasks, we reveal that VLMs exhibit a localization-without-utilization regime: evidence-localizing attention emerges sharply in the middle-to-late layers and is largely decoupled from answer correctness, yet simply enlarging the localized spans on the rendered page recovers a large fraction of the failures. Building on these observations, we propose AGAR (Attention-Guided Adaptive Rendering), a training-free, model-agnostic method that leverages a VLM's own middle-to-late layer attention to identify the top-K important visual patches, maps them back to word spans, and re-renders the page with those spans enlarged before re-inferring the answer. Extensive experiments across nine VTC benchmarks (short-form, long-context, and multi-page memory QA) and four VLM backbones show that AGAR (i)consistently improves off-the-shelf VLMs as a plug-and-play enhancement, (ii)composes with VLM post-training to yield further gains, and (iii)remains robust under both visual- and text-side input degradation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。