用轻量模块提升多模态模型对图像细节的理解能力。
MoDA: Modulation Adapter for Fine-Grained Visual Grounding in Instructional MLLMs
- 通过指令引导的通道级调制，精准控制视觉特征响应
- 在12个基准上均提升性能，最高增益达+12.0分
- 无需修改架构，计算开销小于1%浮点运算
多模态大语言模型在遵循指令任务中表现优异，但因视觉补丁表示存在语义纠缠，难以精准定位图像中的细节。为此，本文提出轻量级模块MoDA（Modulation Adapter），通过指令引导的通道级乘性调制，增强视觉接地能力。与基于补丁的加性选择方法不同，MoDA在已对齐的特征上进行动态调制，实现对相关嵌入维度的细粒度控制。采用标准LLaVA训练流程，在语言指令与预对齐视觉特征间施加交叉注意力，生成动态调制掩码，无需额外监督或结构改动。在涵盖视觉问答、视觉推理和幻觉检测的12个基准上评估，包括2024年新基准MMVP、CV-Bench、MMStar、RealWorldQA，覆盖三种主流架构：LLaVA-1.5、LLaVA-MoRE（2025）和Qwen3-VL（2025）。MoDA在三类模型上均取得一致提升，其中在LLaVA-1.5家族上达到+12.0分（MMVP），LLaVA-MoRE家族+4.8分（ScienceQA），Qwen3-VL上分别提升+4.9（ScienceQA）、+4.1（RealWorldQA）、+3.8（GQA），验证了其泛化能力且仅引入<1%浮点运算开销。代码已开源。
原文摘要 · Abstract (English)
Multimodal Large Language Models (MLLMs) have achieved remarkable success in instruction-following tasks by integrating pretrained visual encoders with large language models (LLMs). However, existing approaches often struggle with fine-grained visual grounding due to semantic entanglement in visual patch representations, where individual patches blend multiple distinct visual elements, making it difficult for models to focus on instruction-relevant details. To address this challenge, we propose MoDA (Modulation Adapter), a lightweight module that enhances visual grounding through instruction-guided channel-wise modulation. Unlike token-level methods such as Q-Former that perform additive feature selection, MoDA operates at the channel level through multiplicative modulation on already-aligned features, enabling fine-grained control over which embedding dimensions are relevant for each instruction. Following the standard LLaVA training protocol, MoDA applies cross-attention between language instructions and pre-aligned visual features, generating dynamic modulation masks without architectural modifications or additional supervision. We evaluate MoDA across 12 benchmarks spanning visual question answering, vision-centric reasoning, and hallucination detection, including recent 2024 benchmarks (MMVP, CV-Bench, MMStar, RealWorldQA), on three distinct MLLM architectures: LLaVA-1.5, LLaVA-MoRE (2025), and Qwen3-VL (2025). MoDA delivers consistent gains across all three families, with +12.0 points on MMVP for the LLaVA-1.5 family and +4.8 points on ScienceQA for the LLaVA-MoRE family, and +4.9 ScienceQA, +4.1 RealWorldQA, and +3.8 GQA on Qwen3-VL, confirming that the gains generalize beyond CLIP-based encoders with minimal overhead (<1% FLOPs). Code is available at https://github.com/waybarrios/MoDA.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。