用大模型实现无需类别标注的物体计数,大幅降低人工标注成本。
Bootstrapping MLLM for Weakly-Supervised Class-Agnostic Object Counting
- 通过对话式分解和对比排序策略,引导大模型逐步估算物体数量。
- 在多个数据集上达到或超越全监督方法性能,平均误差低于10%。
- 适合需要快速部署且标注资源有限的计数场景,如交通监控、疫情统计。
物体计数是计算机视觉中的基础任务,广泛应用于诸多现实场景。全监督方法需为每个物体提供点级标注,成本高昂。少数弱监督方法仅依赖图像级物体数量作为监督信号,已取得较好效果,但通常仅限于单一类别(如人)。本文提出首个基于多模态大模型(MLLM)的弱监督无类别物体计数框架——WS-COC。不同于直接微调MLLM预测数量,我们设计三种有效策略:首先,采用分而辨析对话调优策略,通过多轮对话引导模型判断数量范围并逐步细化;其次,引入对比排序优化策略,训练模型按相对数量对多张图像进行排序;第三,采用全局-局部融合增强策略,整合局部与全局预测以提升密集场景下的计数精度。在FSC-147、CARPK、PUCPR+和ShanghaiTech等数据集上的大量实验表明,WS-COC在显著降低标注成本的同时,性能可媲美甚至超越多项先进全监督方法。代码已开源:https://github.com/viscom-tongji/WS-COC。
原文摘要 · Abstract (English)
Object counting is a fundamental task in computer vision, with broad applicability in many real-world scenarios. Fully-supervised counting methods require costly point-level annotations per object. Few weakly-supervised methods leverage only image-level object counts as supervision and achieve fairly promising results. They are, however, often limited to counting a single category, e.g. person. In this paper, we propose WS-COC, the first MLLM-driven weakly-supervised framework for class-agnostic object counting. Instead of directly fine-tuning MLLMs to predict object counts, which can be challenging due to the modality gap, we incorporate three simple yet effective strategies to bootstrap the counting paradigm in both training and testing: First, a divide-and-discern dialogue tuning strategy is proposed to guide the MLLM to determine whether the object count falls within a specific range and progressively break down the range through multi-round dialogue. Second, a compare-and-rank count optimization strategy is introduced to train the MLLM to optimize the relative ranking of multiple images according to their object counts. Third, a global-and-local counting enhancement strategy aggregates and fuses local and global count predictions to improve counting performance in dense scenes. Extensive experiments on FSC-147, CARPK, PUCPR+, and ShanghaiTech show that WS-COC matches or even surpasses many state-of-art fully-supervised methods while significantly reducing annotation costs. Code is available at https://github.com/viscom-tongji/WS-COC.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。