无需提示即可追踪视频中所有运动并生成自由描述,定位精确到轨迹点。
Track and Caption Any Motion: Open-Vocabulary Spatiotemporal Captioning via Trajectory-Conditioned Generation
- 通过轨迹条件生成,在像素级点粒度上联合跟踪与语言生成。
- 单次推理完成全视频事件描述,包含时间区间与对应轨迹指针。
- 仅用分割标注训练,不依赖额外事件标注,适合多主体动态场景理解。
我们提出TCAM(Track and Caption Any Motion),一种生成式框架,可在无文本查询、无区域提示的情况下观看视频,自动识别运动物体,以开放词汇描述每项运动,精确定位其时间范围,并指向确切的运动轨迹。现有密集点追踪器可实现亚物体级精度但不输出语言,而视频-语言模型需依赖查询且仅使用片段级特征,无法分辨具体移动像素;对象级描述器虽缩小差距,但仍基于检测框或掩码,未触及个体轨迹。TCAM通过一个‘命名感知重采样器’,将少量可学习查询与密集点轨迹标记交叉注意力,提炼为固定长度的运动上下文,驱动语言解码器生成全视频事件。解码器单次前向传播完成所有事件生成,每个事件含自由形式描述、起止时间及对应的轨迹指针,支持多个主体同时活跃的序列事件。训练仅利用现有分割标注,无需额外事件标注,监督描述质量、指针-掩码对齐与指针多样性。在超过5万段视频上,TCAM超越密集视频描述基线,媲美专用查询引导定位与点追踪方法,尽管不使用任何查询,表明轨迹条件生成是运动驱动视频理解的直接路径。
原文摘要 · Abstract (English)
We present TCAM (Track and Caption Any Motion), a generative framework that watches a video and with no text query and no region prompt decides what is moving, describes each motion in open vocabulary, locates it in time, and points to the exact trajectories that carry it. Two mature lines of work make this possible yet leave it unsolved: dense point trackers follow pixels with sub-object precision but emit no language, while video-language models produce fluent descriptions only when handed a query and only from clip-level features that cannot resolve which pixels move. Object-level captioners narrow the gap but still reason over detector boxes or masks, never reaching individual trajectories. TCAM couples tracking and language at point granularity through a Caption-Aware Resampler, where a small set of learnable queries cross-attends to dense point trajectory tokens and distills them into a fixed-length motion context that conditions a language decoder. The decoder generates an entire video's events in a single pass, each with a free-form caption, a start and end time, and a pointer to the trajectories it refers to, for sequential events and several subjects active at once. Training uses only existing segmentation annotations, with no extra event labeling, to supervise caption quality, pointer-mask alignment, and pointer diversity. On over 50K clips, TCAM outperforms dense video captioning baselines and matches dedicated, query-based grounding and point-tracking methods despite using no query, showing that trajectory-conditioned generation is a direct route to motion-driven video understanding.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。