改进视觉注意力分配，让大模型更准地理解图文内容。
RAVE: Re-Allocating Visual Attention in Large Multimodal Models
- 用轻量级门控机制重新分配视觉注意力，提升跨模态对齐。
- 在多语言文字识别等任务上平均提升3分，关键任务增益显著。
- 无需改动主干网络，可端到端训练，适合部署到现有模型。
大型多模态模型（LMMs）继承了预训练语言模型的自注意力机制，但标准注意力存在分配不佳的问题，包括文本与视觉证据之间的跨模态错配，以及视觉标记内部的不均衡。本文提出RAVE（Re-Allocating Visual Attention），一种轻量级的成对门控机制，在视觉键的预软化注意力分数中加入由预罗佩查询和键特征推导出的学习查询-键偏置。RAVE无需修改骨干网络结构，可与模型其余部分端到端训练。在一系列多模态基准测试中，相较于标准注意力，RAVE平均提升3个百分点，尤其在感知密集型任务上表现突出——包括多语言OCR、图表理解、文档VQA和场景文本VQA，这些任务对精确视觉定位至关重要。
原文摘要 · Abstract (English)
Large multimodal models (LMMs) inherit the self-attention mechanism of pretrained language backbones, yet standard attention can exhibit suboptimal allocation, including cross-modal misallocation between textual and visual evidence and intra-visual imbalance among visual tokens. We propose RAVE (Re-Allocating Visual Attention), a lightweight pair-gating mechanism that adds a learned query-key bias to pre-softmax attention scores over visual keys, derived from pre-RoPE query and key features. RAVE requires no architectural modification to the backbone and can be trained end-to-end with the rest of the model. Across a suite of multimodal benchmarks, RAVE improves over standard attention by an average of 3 points, with the largest gains on perception-intensive tasks -- including multilingual OCR, chart understanding, document VQA, and scene text VQA -- where accurate visual grounding is critical.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。