动态分配视觉定位计算资源,提升复杂图像理解效率与精度。
Focus When Necessary: Adaptive Routing and Collaborative Grounding for Training-Free Visual Grounding

- 根据查询难度自适应选择是否处理图像,避免无效计算。
- 在困难样本上通过双阶段精炼实现小目标或遮挡目标的精准定位。
- 无需训练,推理延迟更低,适合对速度敏感的应用场景。
尽管多模态大语言模型在跨模态推理方面表现优异,但在复杂高分辨率图像中仍难以捕捉细粒度细节。现有免训练方法通过图像缩放和局部裁剪来改善,但盲目应用会引入计算冗余,或因截断全局上下文、引入无关背景噪声而降低准确率。为此,我们提出懒惰式多阶段思维(LazyMCoT),一种动态且免训练的框架,依据样本难度自适应分配视觉定位资源。该框架包含自适应路由机制,仅需一次前向传播即可利用首个词元统计量评估预测不确定性,高效跳过置信度高的样本,同时通过共形校准确保困难样本的召回率。对于困难样本,协同定位模块结合模型内在跨模态注意力与外部视觉专家,通过两阶段精炼过程生成精确的局部化输出,以恢复小目标或被遮挡目标。在多个基准上的大量实验表明,LazyMCoT在保持推理准确性的同时,显著降低平均推理延迟,性能媲美训练型方法。代码已开源:https://github.com/TencentBAC/LazyMCoT。
原文摘要 · Abstract (English)
While Multimodal Large Language Models (MLLMs) excel in cross-modal reasoning, they often struggle to perceive fine-grained details in complex high-resolution images. Recent training-free methods address this through image scaling and localized cropping. However, applying these manipulations indiscriminately introduces computational redundancy for simple queries and can degrade accuracy by truncating essential global context or introducing irrelevant background noise. To this end, we propose LazyMCoT, a dynamic and training-free framework that adaptively allocates visual grounding efforts based on sample difficulty. The framework features an Adaptive Routing mechanism that evaluates predictive uncertainty using first-token statistics from a single forward pass. This efficiently bypasses confident cases while ensuring the recall of difficult samples via conformal calibration. For these challenging cases, a Collaborative Grounding module integrates the inherent cross-modal attention of the model with an external visual expert through a two-stage refinement process. This refinement process generates a precise localized display to recover small or occluded targets. Extensive experiments across diverse benchmarks demonstrate that LazyMCoT rivals training-based approaches by simultaneously improving reasoning accuracy and reducing average inference latency. Our code is availble at https://github.com/TencentBAC/LazyMCoT.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。