arXiv:2607.09008cs.CV2026-07

通过动态优化提示词,让视觉语言模型更好识别罕见目标。

C-GAP: Class-Aware and Online Prompting Improves Vision-Language Models on Imbalanced Classes

论文配图:C-GAP: Class-Aware and Online Prompting Improves Vision-Language Models on Imbalanced Classes
图 1 · 摘自论文原文
  • 用复合描述+类别数量信息构建初始提示,提升基础检测性能。
  • 利用大模型迭代优化提示,使少数类[email protected]提升最高达81%。
  • 无需训练或标注,适合安全关键场景中稀有目标检测。

安全关键感知系统需在小标签空间中可靠检测稀有目标,而传统长尾检测方法针对数百个密集标注类别设计,不适用。开放词汇检测器通过推理时使用自然语言查询,使提示质量成为影响性能的关键因素。本文提出C-GAP(Caption-Guided Augmentation and Prompting),一种检测器无关、无需标注的在线提示优化框架。第一阶段构建融合图像场景描述与类别数量信息的复合提示基线,其性能优于仅用场景描述或仅用类别数量的提示。第二阶段由大模型逐图像迭代优化提示,根据少数类[email protected]与动态阈值的对比,将尝试分为接受、暂存或重试三类,一旦达到足够增益即终止。全程不更新检测器权重。实验表明,C-GAP将少数类平均精度提升最高达53%;在COCO上,少数类[email protected]相对基线提升约81%(17.69 → 32.09)。结果证实复合提示是有效优化的基础:以单一提示类型为起点会导致收益递减,说明两阶段设计均为必要贡献。

原文摘要 · Abstract (English)

Safety-critical perception systems must reliably detect rare object classes within small label spaces, a setting that long-tailed detection methods, designed for hundreds of classes with dense annotation, fundamentally do not address. Open-vocabulary detectors offer a promising alternative, as they use natural language queries at inference time, making prompt quality a first-class lever for detection performance. We exploit this property to address class imbalance: rather than retraining models or collecting additional annotations, we ask whether iteratively refining the language prompts, fed to frozen detectors, can improve minority class detection. We introduce C-GAP Caption-Guided Augmentation and Prompting), a detector-agnostic, annotation-free framework that operates in two phases. First, we establish a composite caption baseline combining per-image scene descriptions with class-quantity context, which we show outperforms scene-description only or class-quantity-only prompts across multiple open-vocabulary architectures and benchmarks. Second, an LLM iteratively refines each image's caption individually, with trials triaged into accept, tentative, or regenerate buckets based on minority-class [email protected] against a dynamic threshold derived from the composite baseline. Refinement terminates early once sufficient [email protected] gain is achieved. No detector weights are updated at any stage. Our experiments shows that C-GAP improves minority-class average precision up to 53% over the baselines. On COCO, C-GAP improves minority-class [email protected] by ~81% relative over the composite baseline (17.69 -> 32.09). Experiments confirm that composite captions provide the critical foundation for effective refinement: using scene-description-only or class-quantity-only prompts as the refinement starting point yields diminishing returns, supporting both stages of C-GAP as necessary contributions.

视觉语言模型提示优化长尾检测开放词汇

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。