通过熵值引导裁剪与注意力正则化，提升视觉语言模型的鲁棒性。
ENCORE: Entropy-Guided Cropping and Attention Regularization for Robust Vision--Language Understanding
- 基于跨模态注意力熵选择高相关区域裁剪图像
- 训练时加入熵正则项，强化关键视觉标记关注
- 仅微调0.14%参数，实现平均1.43%准确率提升
视觉语言模型在多任务中表现优异，但基于Transformer的视觉编码器将图像分割为固定分辨率子图，导致轻量级模型中物体完整性受损。现有方法仅关注视觉模态，无法动态保留与提示相关的区域，限制性能。本文发现早期层图像-文本熵与答案定位质量及任务准确率强相关。基于此，提出ENCORE框架：推理阶段采用熵值引导裁剪策略（ECS），评估候选裁剪区域并选择熵最小者以保留与提示相关的连续区域；训练阶段引入熵正则化训练（ERT），在下一词预测基础上增加熵项，增强对关键视觉标记的关注，抑制无关标记。在十个VQA基准测试中，ENCORE仅微调0.14%参数，平均提升1.43%准确率，在近期20亿参数量级视觉语言模型中达到领先水平。代码已开源。
原文摘要 · Abstract (English)
Vision-Language Models (VLMs) perform well on diverse vision-language tasks, but transformer-based visual encoders split images into fixed-resolution sub-images, compromising object integrity in lightweight VLMs. Existing methods only focus on the visual modality and fail to dynamically preserve the integrity of prompt-relevant regions, limiting performance. In this work, we observe that the early-layer image-text entropy of cross-modal attention strongly correlates with answer grounding quality and task accuracy. Building on this finding, we propose \textbf{ENCORE}, an entropy-guided framework with two components: At inference, an \textbf{Entropy-based Cropping Strategy} (ECS) evaluates a small set of candidate crops and selects the one with minimal entropy, preserving contiguous regions relevant to the prompt. At training, \textbf{Entropy Regularization Training} (ERT) augments next-token prediction with an entropy term that sharpens attention on key visual tokens while down-weighting irrelevant ones. Experiments on ten VQA benchmarks show that ENCORE, fine-tuning only 0.14\% of parameters, achieves an average 1.43\% accuracy gain and state-of-the-art performance among recent 2B-parameter VLMs. Our code is released in https://github.com/baokou-fw2/ENCORE.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。