提出全景描述新任务，用最少文字完整还原图像内容。
Panoptic Captioning: An Equivalence Bridge for Image and Text
- 将图像描述分解为实体、位置、属性、关系和全局状态的分步生成。
- 自建数据引擎和评测指标，模型性能超越GPT-4o等主流大模型。
- 适合需要精准图文对齐与多粒度理解的研究者与应用开发者。
本文提出全景描述（Panoptic Captioning）这一新任务，旨在寻找图像的最小文本等价表达，具有广泛应用潜力。我们首次将其定义为生成涵盖所有实体、其位置与属性、实体间关系及全局图像状态的综合性文本描述。通过广泛评估发现，现有先进多模态大模型在该任务上表现有限。为此，我们提出PancapEngine数据引擎，通过精细检测套件识别图像中多样实体，并利用实体感知提示生成高质量标注；同时提出PancapChain方法，将复杂任务分解为多个阶段逐步生成。更重要的是，我们构建了全新评测指标PancapScore和人工校验测试集以实现可靠评估。实验表明，我们的PancapChain-13B模型在性能上超越开源模型InternVL-2.5-78B，甚至优于闭源模型GPT-4o和Gemini-2.0-Pro，验证了数据与方法的有效性。
原文摘要 · Abstract (English)
This work introduces panoptic captioning, a novel task striving to seek the minimum text equivalent of images, which has broad potential applications. We take the first step towards panoptic captioning by formulating it as a task of generating a comprehensive textual description for an image, which encapsulates all entities, their respective locations and attributes, relationships among entities, as well as global image state. Through an extensive evaluation, our work reveals that state-of-the-art Multi-modal Large Language Models (MLLMs) have limited performance in solving panoptic captioning. To address this, we propose an effective data engine named PancapEngine to produce high-quality data and a novel method named PancapChain to improve panoptic captioning. Specifically, our PancapEngine first detects diverse categories of entities in images by an elaborate detection suite, and then generates required panoptic captions using entity-aware prompts. Additionally, our PancapChain explicitly decouples the challenging panoptic captioning task into multiple stages and generates panoptic captions step by step. More importantly, we contribute a comprehensive metric named PancapScore and a human-curated test set for reliable model evaluation. Experiments show that our PancapChain-13B model can beat state-of-the-art open-source MLLMs like InternVL-2.5-78B and even surpass proprietary models like GPT-4o and Gemini-2.0-Pro, demonstrating the effectiveness of our data engine and method. Project page: https://visual-ai.github.io/pancap/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。