让用户用框选物体,实现精准视频分割与描述生成。
Scene Graph-guided SegCaptioning Transformer with Fine-grained Alignment for Controllable Video Segmentation and Captioning
- 通过提示图结构引导细粒度对齐,融合用户意图
- 在两个数据集上实现高精度掩码与文本联合生成
- 适合需要精细控制视频内容理解的交互应用
近年来多模态大模型显著弥合了不同模态间的表征差距,推动了视频多模态理解的发展,通过生成相关模态提升用户对视频内容的理解。然而,现有方法多关注全局理解,缺乏用户交互能力。为此,本文提出新任务——可控视频分割与描述生成(SegCaptioning),允许用户通过特定提示(如框选感兴趣物体)同时生成精确的掩码和描述,以准确体现用户意图。我们设计了场景图引导的细粒度分割-描述变换器(SG-FSCFormer),引入提示引导的时间图网络,通过自适应提示适配器有效捕捉并表示用户意图。此外,模型采用细粒度掩码-语言解码器,结合多实体对比损失,协同预测高质量的图文对,并实现每个掩码与对应文本标记间的细粒度对齐,从而增强用户对视频的理解。在两个基准数据集上的全面实验表明,该模型能有效捕捉用户意图,生成符合用户需求的精准多模态输出。代码已开源:https://github.com/XuZhang1211/SG-FSCFormer。
原文摘要 · Abstract (English)
Recent advancements in multimodal large models have significantly bridged the representation gap between diverse modalities, catalyzing the evolution of video multimodal interpretation, which enhances users' understanding of video content by generating correlated modalities. However, most existing video multimodal interpretation methods primarily concentrate on global comprehension with limited user interaction. To address this, we propose a novel task, Controllable Video Segmentation and Captioning (SegCaptioning), which empowers users to provide specific prompts, such as a bounding box around an object of interest, to simultaneously generate correlated masks and captions that precisely embody user intent. An innovative framework Scene Graph-guided Fine-grained SegCaptioning Transformer (SG-FSCFormer) is designed that integrates a Prompt-guided Temporal Graph Former to effectively captures and represents user intent through an adaptive prompt adaptor, ensuring that the generated content well aligns with the user's requirements. Furthermore, our model introduces a Fine-grained Mask-linguistic Decoder to collaboratively predict high-quality caption-mask pairs using a Multi-entity Contrastive loss, as well as provide fine-grained alignment between each mask and its corresponding caption tokens, thereby enhancing users' comprehension of videos. Comprehensive experiments conducted on two benchmark datasets demonstrate that SG-FSCFormer achieves remarkable performance, effectively capturing user intent and generating precise multimodal outputs tailored to user specifications. Our code is available at https://github.com/XuZhang1211/SG-FSCFormer.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。