提出双向轨迹对齐框架，提升视频推理分割精度
Learning Trajectory-Aware Multimodal Large Language Models for Video Reasoning Segmentation
- 引入双向文本-轨迹对齐机制，增强模型对物体运动轨迹感知
- 在多个数据集上超越现有方法，最高提升6.2% mIoU
- 适合需要精准视频目标分割的视觉语言任务研究者
多模态大语言模型（MLLMs）的发展推动了视频推理分割的需求，即根据人类指令分割视频中的对象。以往研究依赖单向且隐式的文本-轨迹对齐，在严重视频动态情况下难以准确感知轨迹。本文提出TrajSeg，一个基于MLLMs的统一框架。具体而言，引入双向文本-轨迹对齐，使MLLMs接收以定位为目的（文本到轨迹）和以描述为目的（轨迹到文本）的指令。这使得模型能获得更强对应关系，并更好感知视频中物体的运动轨迹。通过帧级内容融合（FCI）模块将MLLM解析的轨迹级标记适配为帧特定信息，再由统一掩码解码器完成所有帧的分割，实现简化且端到端可训练。在引用与推理视频分割数据集上的大量实验表明，TrajSeg在所有指标上均优于现有方法。代码将公开于https://github.com/haodi19/TrajSeg。
原文摘要 · Abstract (English)
The prosperity of Multimodal Large Language Models (MLLMs) has stimulated the demand for video reasoning segmentation, which aims to segment video objects based on human instructions. Previous studies rely on unidirectional and implicit text-trajectory alignment, which struggles with trajectory perception when faced with severe video dynamics. In this work, we propose TrajSeg, a simple and unified framework built upon MLLMs. Concretely, we introduce bidirectional text-trajectory alignment, where MLLMs accept grounding-intended (text-to-trajectory) and captioning-intended (trajectory-to-text) instructions. This way, MLLMs can benefit from enhanced correspondence and better perceive object trajectories in videos. The mask generation from trajectories is achieved via a frame-level content integration (FCI) module and a unified mask decoder. The former adapts the MLLM-parsed trajectory-level token to frame-specific information. The latter unifies segmentation for all frames into a single structure, enabling the proposed framework to be simplified and end-to-end trainable. Extensive experiments on referring and reasoning video segmentation datasets demonstrate the effectiveness of TrajSeg, which outperforms all video reasoning segmentation methods on all metrics. The code will be publicly available at https://github.com/haodi19/TrajSeg.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。