提升视频理解中目标定位的精准与稳定，解决移动物体跟踪漂移问题。
SPARROW: Learning Spatial Precision and Temporal Referential Consistency in Pixel-Grounded Video MLLMs
- 用时序对齐特征注入训练，实现对象跨帧精准追踪。
- 双提示设计融合框与分割信息，定位精度提升至mIoU+5。
- 无需外部检测器，适配主流视频多模态模型，通用性强。
多模态大模型已从图像级推理发展到像素级定位，但视频理解仍面临空间精度与时间一致性双重挑战。现有视频多模态模型常依赖静态分割标记（[SEG]）进行逐帧定位，缺乏时间上下文，导致对象移动或重现身时出现定位漂移、身份混淆和初始化不稳。本文提出SPARROW，一种像素级视频多模态大模型，通过两个核心组件统一提升空间精度与时间稳定性：(i) 目标特定追踪特征（TSF），在训练中注入时序对齐的指代线索；(ii) 双提示设计，解码框([BOX])与分割([SEG])标记，融合几何先验与语义定位。SPARROW基于30,646段视频与45,231个问答对的标注数据集构建，采用无类别依赖的SAM2-based提案器，端到端运行且无需外部检测器。集成至UniPixel、GLUS与VideoGLaMM三个开源视频多模态模型后，在六个基准上均取得一致提升，最高达RVOS上+8.9 J&F、视觉定位+5 mIoU、GCG上+5.4 CLAIR。结果表明，SPARROW显著提升了像素级视频理解中的指代稳定性、空间精度与时间连贯性。
原文摘要 · Abstract (English)
Multimodal large language models (MLLMs) have advanced from image-level reasoning to pixel-level grounding, but extending these capabilities to videos remains challenging as models must achieve spatial precision and temporally consistent reference tracking. Existing video MLLMs often rely on a static segmentation token ([SEG]) for frame-wise grounding, which provides semantics but lacks temporal context, causing spatial drift, identity switches, and unstable initialization when objects move or reappear. We introduce SPARROW, a pixel-grounded video MLLM that unifies spatial accuracy and temporal stability through two key components: (i) Target-Specific Tracked Features (TSF), which inject temporally aligned referent cues during training, and (ii) a dual-prompt design that decodes box ([BOX]) and segmentation ([SEG]) tokens to fuse geometric priors with semantic grounding. SPARROW is supported by a curated referential video dataset of 30,646 videos and 45,231 Q&A pairs and operates end-to-end without external detectors via a class-agnostic SAM2-based proposer. Integrated into three recent open-source video MLLMs (UniPixel, GLUS, and VideoGLaMM), SPARROW delivers consistent gains across six benchmarks, improving up to +8.9 J&F on RVOS, +5 mIoU on visual grounding, and +5.4 CLAIR on GCG. These results demonstrate that SPARROW substantially improves referential stability, spatial precision, and temporal coherence in pixel-grounded video understanding. Project page: https://risys-lab.github.io/SPARROW
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。