通过强化图文对齐,让零样本计数模型准确识别未见类别物体。
Expanding Zero-Shot Object Counting with Rich Prompts
- 用文本-图像相似性训练的编码器增强提示词表征。
- 在三个数据集上实现当前最佳零样本计数性能。
- 适合需要跨类别泛化的开放世界计数场景使用。
将预训练的零样本计数模型扩展到未见类别,不能仅靠增加新提示词,因为这无法实现文本与视觉特征间的有效对齐。本文提出RichCount框架,采用两阶段训练策略,提升文本编码并强化模型对图像中物体的关联能力。该方法包含两个关键目标:(1) 利用前馈网络和适配器,基于文本-图像相似性训练,丰富文本特征表示,构建鲁棒且对齐的表征;(2) 将优化后的编码器应用于计数任务,实现对多样化提示词和复杂图像的有效泛化。通过建立有意义的特征对齐,RichCount突破了简单提示扩展的局限,显著提升未见类别的计数准确性。在三个基准数据集上的大量实验表明,该方法达到零样本计数的最先进性能,并大幅增强开放世界场景下对未见类别的泛化能力。
原文摘要 · Abstract (English)
Expanding pre-trained zero-shot counting models to handle unseen categories requires more than simply adding new prompts, as this approach does not achieve the necessary alignment between text and visual features for accurate counting. We introduce RichCount, the first framework to address these limitations, employing a two-stage training strategy that enhances text encoding and strengthens the model's association with objects in images. RichCount improves zero-shot counting for unseen categories through two key objectives: (1) enriching text features with a feed-forward network and adapter trained on text-image similarity, thereby creating robust, aligned representations; and (2) applying this refined encoder to counting tasks, enabling effective generalization across diverse prompts and complex images. In this manner, RichCount goes beyond simple prompt expansion to establish meaningful feature alignment that supports accurate counting across novel categories. Extensive experiments on three benchmark datasets demonstrate the effectiveness of RichCount, achieving state-of-the-art performance in zero-shot counting and significantly enhancing generalization to unseen categories in open-world scenarios.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。