arXiv:2510.02898cs.CV2025-10被引 1

用图像块生成描述,无需标注即可任意区域生成文本。

One Patch to Caption Them All: A Unified Zero-Shot Captioning Framework

  • 以图像块为单位,而非整图,实现灵活区域描述。
  • 在多个区域标注任务中超越现有方法,最高提升3.2个CIDEr分数。
  • 适合需要零样本、灵活生成的视觉理解场景。

零样本图像描述模型通常依赖全局图像特征和成对图文数据来生成描述。本文提出一种统一的零样本描述框架,从图像中心转向块中心范式,无需区域级标注即可描述任意区域。我们把图像块作为基本描述单元,通过聚合实现从单个块到不连续区域乃至整图的描述。研究发现,生成有意义密集视觉特征的主干网络(如DINO)是该框架取得最优性能的关键。实验表明,相比其他基线与先进方法,本模型在零样本密集描述和区域集描述任务上表现更优,且引入新的可追踪描述任务,进一步验证了块级语义表示在灵活生成中的有效性。

原文摘要 · Abstract (English)

Zero-shot captioners are recently proposed models that utilize common-space vision-language representations to caption images without relying on paired image-text data. To caption an image, they proceed by textually decoding a text-aligned image feature, but they limit their scope to global representations and whole-image captions. We present a unified framework for zero-shot captioning that shifts from an image-centric to a patch-centric paradigm, enabling the captioning of arbitrary regions without the need of region-level supervision. Instead of relying on global image representations, we treat individual patches as atomic captioning units and aggregate them to describe arbitrary regions, from single patches to non-contiguous areas and entire images. We analyze the key ingredients that enable current latent captioners to work in our novel proposed framework. Experiments demonstrate that backbones producing meaningful, dense visual features, such as DINO, are key to achieving state-of-the-art performance in multiple region-based captioning tasks. Compared to other baselines and state-of-the-art competitors, our models achieve better performance on zero-shot dense captioning and region-set captioning. We also introduce a new trace captioning task that further demonstrates the effectiveness of patch-wise semantic representations for flexible caption generation. Project page at https://paciosoft.com/Patch-ioner/ .

零样本图像描述块级生成

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。