首个兼顾实例感知的多任务视觉定位框架,统一细粒度与粗粒度预测。
Improving Generalized Visual Grounding with Instance-aware Joint Learning
- 用实例查询统一框和掩码的联合预测,确保多粒度一致性。
- 在10个数据集上达到领先性能,显著超越现有方法。
- 适合需要精准定位与跨粒度对齐的研究者或工业应用。
广义视觉定位任务(包括广义指代表达理解GREC和分割GRES)通过引入多目标与非目标场景,扩展了经典视觉定位范式。其中,GREC关注粗粒度边界框级别的准确识别,而GRES则追求像素级精细感知。然而,现有方法通常独立处理这两项任务,忽略了联合训练带来的多粒度预测一致性与流程简化优势。此外,当前方法常将GRES视为语义分割任务,忽视了实例感知能力的重要性及实例级框与掩码间的一致性需求。为此,我们提出InstanceVG,一种具备实例感知能力的多任务广义视觉定位框架,利用实例查询统一实例级框与掩码的联合预测与一致性。据我们所知,InstanceVG是首个同时解决GREC与GRES并融入实例感知机制的框架。为实现该框架,我们为每个实例查询分配一个先验参考点,该点亦作为目标匹配的额外依据,从而实现同一实例在点、框、掩码上的预测一致性。在四个任务的十个数据集上的大量实验表明,InstanceVG在多种评估指标上均达到最先进水平,显著优于现有方法。代码与模型将公开于https://github.com/Dmmm1997/InstanceVG。
原文摘要 · Abstract (English)
Generalized visual grounding tasks, including Generalized Referring Expression Comprehension (GREC) and Segmentation (GRES), extend the classical visual grounding paradigm by accommodating multi-target and non-target scenarios. Specifically, GREC focuses on accurately identifying all referential objects at the coarse bounding box level, while GRES aims for achieve fine-grained pixel-level perception. However, existing approaches typically treat these tasks independently, overlooking the benefits of jointly training GREC and GRES to ensure consistent multi-granularity predictions and streamline the overall process. Moreover, current methods often treat GRES as a semantic segmentation task, neglecting the crucial role of instance-aware capabilities and the necessity of ensuring consistent predictions between instance-level boxes and masks. To address these limitations, we propose InstanceVG, a multi-task generalized visual grounding framework equipped with instance-aware capabilities, which leverages instance queries to unify the joint and consistency predictions of instance-level boxes and masks. To the best of our knowledge, InstanceVG is the first framework to simultaneously tackle both GREC and GRES while incorporating instance-aware capabilities into generalized visual grounding. To instantiate the framework, we assign each instance query a prior reference point, which also serves as an additional basis for target matching. This design facilitates consistent predictions of points, boxes, and masks for the same instance. Extensive experiments obtained on ten datasets across four tasks demonstrate that InstanceVG achieves state-of-the-art performance, significantly surpassing the existing methods in various evaluation metrics. The code and model will be publicly available at https://github.com/Dmmm1997/InstanceVG.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。