无需训练，让大模型像人一样动态聚焦视觉重点
DyFo: A Training-Free Dynamic Focus Visual Search for Enhancing LMMs in Fine-Grained Visual Understanding
- 用MCTS模拟人类视觉搜索，双向联动大模型与视觉专家
- 在多个数据集上显著提升细粒度理解能力，降低幻觉率
- 适合需要精准视觉定位的场景，如医疗影像分析
人类能在杂乱环境中轻松定位目标物体，依赖一种称为视觉搜索的认知机制，可高效过滤无关信息并聚焦于任务相关区域。受此启发，我们提出Dyfo（Dynamic Focus），一种无需训练的动态聚焦视觉搜索方法，用于增强大视觉语言模型（LMMs）在细粒度视觉理解中的表现。与需额外模块或数据收集的方法不同，Dyfo通过大模型与视觉专家间的双向交互，利用蒙特卡洛树搜索（MCTS）算法模拟人类般的注意力调整过程。该机制使LMMs能聚焦关键视觉区域，同时过滤无关内容，且无需引入因词汇扩展或专用定位模块带来的额外训练开销。实验结果表明，Dyfo显著提升了细粒度视觉理解性能，并有效缓解了LMMs中的幻觉问题，在固定与动态分辨率模型中均表现优异。代码已公开于https://github.com/PKU-ICST-MIPL/DyFo_CVPR2025。
原文摘要 · Abstract (English)
Humans can effortlessly locate desired objects in cluttered environments, relying on a cognitive mechanism known as visual search to efficiently filter out irrelevant information and focus on task-related regions. Inspired by this process, we propose Dyfo (Dynamic Focus), a training-free dynamic focusing visual search method that enhances fine-grained visual understanding in large multimodal models (LMMs). Unlike existing approaches which require additional modules or data collection, Dyfo leverages a bidirectional interaction between LMMs and visual experts, using a Monte Carlo Tree Search (MCTS) algorithm to simulate human-like focus adjustments. This enables LMMs to focus on key visual regions while filtering out irrelevant content, without introducing additional training caused by vocabulary expansion or the integration of specialized localization modules. Experimental results demonstrate that Dyfo significantly improves fine-grained visual understanding and reduces hallucination issues in LMMs, achieving superior performance across both fixed and dynamic resolution models. The code is available at https://github.com/PKU-ICST-MIPL/DyFo_CVPR2025
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。