通过专家模型与大模型协作,提升细粒度指代理解的准确率与效率。
New Dataset and Methods for Fine-Grained Compositional Referring Expression Comprehension via Specialist-MLLM Collaboration
- 用专家模型定位候选区域,大模型推理选择最合理目标。
- 在新数据集上比单一模型提升12.7%准确率,尤其在复杂场景下表现突出。
- 适合研究多模态推理、视觉语言模型融合的开发者和研究人员。
指代表达理解(REC)是一项基础的跨模态任务,评估语言理解、图像理解及语言到图像定位的协同能力,是检验多模态大模型(MLLMs)的重要基准。为推动该领域发展,我们此前提出一个新数据集,具有两个关键特性:一是可调控难度,需在物体类别、属性和多跳关系上进行多层次细粒度推理;二是包含通过细粒度编辑生成的负向文本和图像,明确测试模型对目标不存在场景的拒识能力,这是现有数据集常忽视却至关重要的挑战。在本扩展工作中,我们提出两种新方法,通过专家模型与MLLM的协作解决细粒度REC难题。第一种方法自适应分配简单样本给轻量级模型,复杂样本交由强大MLLM处理,兼顾准确率与效率;第二种方法由专家模型生成一组可能的目标区域,再由MLLM利用其推理能力选出最合理的选项。这些协作策略在我们的数据集及其他挑战性基准上均取得显著提升。结果表明,结合专用模型与通用模型为解决复杂现实视觉-语言任务提供了可行路径。数据集与代码已开源:https://github.com/sleepyshep/FineCops-Ref。
原文摘要 · Abstract (English)
Referring Expression Comprehension (REC) is a foundational cross-modal task that evaluates the interplay of language understanding, image comprehension, and language-to-image grounding. It serves as an essential testing ground for Multimodal Large Language Models (MLLMs). To advance this field, we introduced a new REC dataset in our previous conference paper, characterized by two key features. First, it is designed with controllable difficulty levels, requiring multi-level fine-grained reasoning across object categories, attributes, and multi-hop relationships. Second, it incorporates negative text and images generated through fine-grained editing and augmentation, explicitly testing a model's ability to reject scenarios where the target object is absent, an often overlooked yet critical challenge in existing datasets. In this extended work, we propose two new methods to tackle the challenges of fine-grained REC by combining the strengths of Specialist Models and MLLMs. The first method adaptively assigns simple cases to faster, lightweight models and reserves complex ones for powerful MLLMs, balancing accuracy and efficiency. The second method lets a specialist generate a set of possible object regions, and the MLLM selects the most plausible one using its reasoning ability. These collaborative strategies lead to significant improvements on our dataset and other challenging benchmarks. Our results show that combining specialized and general-purpose models offers a practical path toward solving complex real-world vision-language tasks. Our dataset and code are available at https://github.com/sleepyshep/FineCops-Ref.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。