让视觉模型更专注关键细节，提升多模态推理准确率
See Less, See Right: Bi-directional Perceptual Shaping For Multimodal Reasoning
- 用双向感知塑造技术，让模型关注问题相关的视觉区域
- 在8个基准上平均提升8.2%，且跨领域泛化能力强
- 适合需要精准视觉理解的多模态任务研究者
大型视觉语言模型（VLMs）通常依赖中间视觉线索，如外部工具注入或推理过程中生成的潜在视觉标记，但这些方法常忽略细粒度视觉证据（如图表中的折线），跨领域泛化能力差，且推理成本高。本文提出双向感知塑造（BiPS），将问题相关的掩码视图转化为双向关注信号，在训练中塑造感知。BiPS首先对原始图像与仅保留问题相关区域的保证据视图施加KL一致性约束，确保支持性像素的粗粒度完整覆盖；随后对原始图像与关键像素被掩码的去证据视图施加KL分离约束，使图像不再支持原答案，从而避免仅依赖文本的捷径，强制细粒度视觉依赖。在8个基准测试中，BiPS使Qwen2.5-VL-7B平均提升8.2%，并在未见数据集和图像类型上表现出强跨域泛化能力。
原文摘要 · Abstract (English)
Large vision-language models (VLMs) often benefit from intermediate visual cues, either injected via external tools or generated as latent visual tokens during reasoning, but these mechanisms still overlook fine-grained visual evidence (e.g., polylines in charts), generalize poorly across domains, and incur high inference-time cost. In this paper, we propose Bi-directional Perceptual Shaping (BiPS), which transforms question-conditioned masked views into bidirectional where-to-look signals that shape perception during training. BiPS first applies a KL-consistency constraint between the original image and an evidence-preserving view that keeps only question-relevant regions, encouraging coarse but complete coverage of supporting pixels. It then applies a KL-separation constraint between the original and an evidence-ablated view where critical pixels are masked so the image no longer supports the original answer, discouraging text-only shortcuts (i.e., answering from text alone) and enforcing fine-grained visual reliance. Across eight benchmarks, BiPS boosts Qwen2.5-VL-7B by 8.2% on average and shows strong out-of-domain generalization to unseen datasets and image types.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。