通过噪声掩码清除冗余视觉信息，提升多模态模型细粒度推理能力
Latent Noise Mask for Reducing Visual Redundancy in Multimodal Large Language Models
- 引入可学习的证据令牌，动态评估视觉标记相关性并保留关键信息
- 在潜在空间注入自适应噪声，使无关视觉内容软抑制，性能提升2.4-6.4点
- 无需修改模型结构，仅用一个控制令牌，适合追求轻量推理优化的研究者
多模态大语言模型在细粒度视觉推理中表现不佳，因问题相关的视觉线索被密集冗余的图像标记稀释。现有方法虽将链式思维扩展至视觉或潜在空间，却忽视了冗余视觉标记的负面影响。本文提出LatEnt Noise maSk（Lens），一种基于问题条件的视觉证据净化框架，使MLLM能在潜在空间中以更清晰的视觉线索进行推理。Lens引入轻量级的Lens Evidence Token（LET），评分并保留与当前问题相关的视觉标记。基于LET得分，向低相关性标记注入自适应潜在噪声，软性抑制干扰项，不改变模型主干或标记序列。仅需一个临时可学习控制令牌和轻量噪声生成器，Lens添加极少开销，在多数VQA数据集上提升2.4–6.4点，在定位任务上提升4.1–6.4点。结果表明，多模态推理更受益于更干净的问题相关视觉证据，而非单纯延长推理链条。
原文摘要 · Abstract (English)
Multimodal large language models (MLLMs) often fail in fine-grained visual reasoning, as question-relevant visual cues are diluted by dense and redundant image tokens. Recent multimodal reasoning methods usually extend chain-of-thought from language models into visual or latent spaces, seeking to add intermediate reasoning states while overlooking the negative impact of redundant visual tokens. We propose LatEnt Noise maSk (Lens), a question-conditioned visual evidence purification framework that empowers MLLMs to reason with cleaner visual cues in latent space. Lens introduces a lightweight Lens Evidence Token (LET) to score which visual tokens support the current question and preserve them during decoding. Guided by the LET scores, it injects adaptive latent noise into low-relevance tokens, softly suppressing distractors without changing the model backbone or token sequence. With only one temporary learnable control token and a lightweight noise generator, Lens adds minimal overhead while improving the base MLLM by 2.4-6.4 points on most VQA datasets and by 4.1-6.4 points on grounding tasks. These results show that multimodal reasoning can benefit more directly from cleaner question-relevant visual evidence than from simply extending the reasoning trace.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。