用注意力引导图像扭曲，让大模型更准看懂细节。
Constructive Distortion: Improving MLLMs with Attention-Guided Image Warping
- 根据模型注意力重分配图像分辨率，聚焦重要区域。
- 在5个数据集上提升准确率，减少幻觉，最高增益达+3.2%。
- 无需改模型，适合所有主流多模态大模型使用。
多模态大语言模型在复杂场景中常忽略小细节和空间关系，导致细粒度感知定位错误。本文提出AttWarp，一种轻量级方法：测试时利用模型跨模态注意力对输入图像进行保形扭曲，将更高分辨率分配给查询相关区域，同时压缩低信息量区域，保持全局上下文。该方法不改变模型权重或架构，仅重新分布原始图像信息。在五个基准（TextVQA、GQA、DocVQA、POPE、MMMU）和四种模型（LLaVA、Qwen-VL、InternVL、InstructBLIP）上，AttWarp持续提升准确率，增强组合推理能力，减少幻觉，优于四个在测试时操作原始图像的竞争基线。结果表明，注意力引导的扭曲能在保留上下文的同时优先处理与查询相关的信息，使相同模型在扭曲输入下表现更优。
原文摘要 · Abstract (English)
Multimodal large language models (MLLMs) often miss small details and spatial relations in cluttered scenes, leading to errors in fine-grained perceptual grounding. We introduce AttWarp, a lightweight method that allocates more resolution to query-relevant content while compressing less informative areas, all while preserving global context. At test time, the approach uses an MLLM's cross-modal attention to perform rectilinear warping of the input image, reallocating spatial resolution toward regions the model deems important, without changing model weights or architecture. This attention-guided warping preserves all original image information but redistributes it non-uniformly, so small objects and subtle relationships become easier for the same model to read while the global layout remains intact. Across five benchmarks (TextVQA, GQA, DocVQA, POPE, MMMU) and four MLLMs (LLaVA, Qwen-VL, InternVL, and InstructBLIP), AttWarp consistently improves accuracy, strengthens compositional reasoning, and reduces hallucinations, outperforming four competitive baselines that manipulate raw images at test time. Together, these results show that attention-guided warping prioritizes information relevant to the query while preserving context, and that the same MLLMs perform better when given such warped inputs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。