无需训练即可精准描述视频中任意物体的动态细节。
Caption Anything in Video: Fine-grained Object-centric Captioning via Spatiotemporal Multimodal Prompting
- 通过时空多模态提示实现对象级细粒度描述
- 支持点、框、不规则区域等多种交互方式
- 适合需要精准视频内容理解的研究与应用
我们提出CAT-V（Caption AnyThing in Video），一种无需训练的细粒度对象中心视频描述框架，可对用户选定对象在时间维度上生成详细描述。该框架集成三个核心组件：基于SAMURAI的分割器实现跨帧精准对象分割，基于TRACE-Uni的时间分析器准确检测事件边界与进行时序分析，以及使用InternVL-2.5的描述生成器。通过时空视觉提示与链式思维推理，框架在不依赖额外训练数据的前提下，生成包含对象属性、动作、状态、交互及环境上下文的详细、时序感知描述。支持点、边界框和不规则区域等多样视觉提示，通过追踪对象状态与交互保持时间敏感性。相比现有方法过于抽象或缺乏对象精度的问题，本方法实现了对象级精细描述，同时保证时序连贯性与空间准确性。项目代码开源于https://github.com/yunlong10/CAT-V。
原文摘要 · Abstract (English)
We present CAT-V (Caption AnyThing in Video), a training-free framework for fine-grained object-centric video captioning that enables detailed descriptions of user-selected objects through time. CAT-V integrates three key components: a Segmenter based on SAMURAI for precise object segmentation across frames, a Temporal Analyzer powered by TRACE-Uni for accurate event boundary detection and temporal analysis, and a Captioner using InternVL-2.5 for generating detailed object-centric descriptions. Through spatiotemporal visual prompts and chain-of-thought reasoning, our framework generates detailed, temporally-aware descriptions of objects' attributes, actions, statuses, interactions, and environmental contexts without requiring additional training data. CAT-V supports flexible user interactions through various visual prompts (points, bounding boxes, and irregular regions) and maintains temporal sensitivity by tracking object states and interactions across different time segments. Our approach addresses limitations of existing video captioning methods, which either produce overly abstract descriptions or lack object-level precision, enabling fine-grained, object-specific descriptions while maintaining temporal coherence and spatial accuracy. The GitHub repository for this project is available at https://github.com/yunlong10/CAT-V
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。