无需训练即可精准控制大模型视觉关注区域。
Test-Time Computing for Referring Multimodal Large Language Models
- 通过可学习的视觉提示，在推理时动态调整模型注意力。
- 在多个测试集上提升区域定位准确率，最高达18.7%。
- 支持框、掩码、涂鸦等多种提示形式，适合视觉精调场景。
我们提出ControlMLLM++，一种新颖的测试时适应框架，通过向冻结的多模态大语言模型（MLLMs）注入可学习的视觉提示，实现无需任何模型重训练或微调的细粒度区域级视觉推理。基于跨模态注意力图天然编码了文本标记与视觉区域之间的语义对应关系这一洞察，ControlMLLM++在推理过程中利用特定任务的能量函数优化一个潜在的视觉标记修正器，以引导模型注意力聚焦于用户指定区域。为增强优化稳定性并缓解语言提示偏差，该方法引入改进的优化策略（Optim++）和提示去偏机制（PromptDebias）。支持多种视觉提示类型，包括边界框、掩码、草图和点。实验表明，该方法具备强泛化能力与良好可解释性。代码已开源：https://github.com/mrwu-mac/ControlMLLM。
原文摘要 · Abstract (English)
We propose ControlMLLM++, a novel test-time adaptation framework that injects learnable visual prompts into frozen multimodal large language models (MLLMs) to enable fine-grained region-based visual reasoning without any model retraining or fine-tuning. Leveraging the insight that cross-modal attention maps intrinsically encode semantic correspondences between textual tokens and visual regions, ControlMLLM++ optimizes a latent visual token modifier during inference via a task-specific energy function to steer model attention towards user-specified areas. To enhance optimization stability and mitigate language prompt biases, ControlMLLM++ incorporates an improved optimization strategy (Optim++) and a prompt debiasing mechanism (PromptDebias). Supporting diverse visual prompt types including bounding boxes, masks, scribbles, and points, our method demonstrates strong out-of-domain generalization and interpretability. The code is available at https://github.com/mrwu-mac/ControlMLLM.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。