统一建模视频中物体的检测、跟踪与描述，提升细粒度理解能力。
CaptionFormer: Unified Segmentation, Tracking, and Captioning for Spatio-Temporal Objects
- 利用视觉语言模型生成合成标注，扩充数据集以支持联合任务训练。
- 在三个基准上达到当前最佳性能，尤其在轨迹描述精度上显著提升。
- 适合研究视频理解、多任务学习与自动内容生成的开发者使用。
密集视频物体描述（DVOC）旨在联合检测、跟踪并为视频中的物体轨迹生成自然语言描述，要求对时空细节有深刻理解。由于任务复杂且人工标注成本高，以往方法受限于数据量，表现受限。为此，我们提出利用先进的视觉语言模型生成时空定位实体的描述，并扩展了LVIS和LV-VIS数据集，新增合成标注版本LVISCap和LV-VISCap。同时，我们设计了一个端到端模型CaptionFormer，可同时完成物体检测、分割、跟踪与描述。该模型在三个现有基准VidSTG、VLN和BenSMOT上取得领先性能。相关数据集与代码已公开：https://www.gabriel.fiastre.fr/captionformer/。
原文摘要 · Abstract (English)
Dense Video Object Captioning (DVOC) is the task of jointly detecting, tracking, and captioning object trajectories in a video, requiring the ability to understand spatio-temporal details and describe them in natural language. Due to the complexity of the task and the high cost associated with manual annotation, previous approaches resort to training strategies with limited data, potentially leading to suboptimal performance. To circumvent this issue, we propose to generate captions about spatio-temporally localized entities leveraging a state-of-the-art VLM, and extend the LVIS and LV-VIS datasets with our synthetic captions (LVISCap and LV-VISCap). Moreover, we introduce an end-to-end model, CaptionFormer, capable of jointly detecting, segmenting, tracking and captioning object trajectories. CaptionFormer achieves state-of-the-art DVOC results on three existing benchmarks, VidSTG, VLN and BenSMOT. The datasets and code are available at https://www.gabriel.fiastre.fr/captionformer/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。