用编码器思路生成高保真视频字幕,减少冗余同时保留细节。
CodecCap: High-Fidelity Codec-Inspired Residual Modeling for Dense Video Captioning

- 借鉴视频编码原理,用关键帧和残差字幕分层表达视频内容。
- 在新构建的VidCapQA评测中,直接生成字幕丢失大量视觉细节。
- 适用于需要精细语义监督的视频理解任务,如复杂动作分析。
现有视频字幕方法难以兼顾视觉保真度与冗余控制:整体式字幕紧凑但丢失细节,分段式字幕覆盖更全却产生大量重复。本文提出CodecCap,一种受视频编码启发的高保真密集视频字幕框架。该框架将视频表示为关键帧字幕与残差字幕的组合:关键帧字幕全面编码稳定的视觉上下文,残差字幕仅捕捉局部动作、运动和变化。这一机制有效保留细粒度视觉证据,同时显著降低冗余描述。为量化字幕保真度,我们引入VidCapQA——一个包含1,000个问题、覆盖14个能力维度的“先生成字幕再问答”基准。实验表明,即使使用强大多模态模型(VLMs)直接生成字幕,仍会遗漏大量视觉信息,凸显字幕表示是关键瓶颈。对比实验显示,CodecCap在相同底层VLM下显著优于直接生成法,证明关键帧-残差字幕结构是实现高保真视频-语言监督的有效路径。此外,我们利用CodecCap构建了CodecVDC-100K数据集,包含锚点、残差、场景级和视频级监督的10万条密集字幕。
原文摘要 · Abstract (English)
Existing video captioning methods struggle to balance visual fidelity and redundancy: holistic captions are compact but lose fine-grained evidence, whereas segment-wise captions improve coverage but introduce heavy redundancy. We propose CodecCap, a codec-inspired framework for high-fidelity dense video captioning. Analogous to video codecs, CodecCap represents videos using keyframe and residual captions. Keyframe captions exhaustively encode stable visual context, while residual captions capture temporally only localized actions, motions and changes. This effectively preserves fine-grained visual evidence while reducing redundant descriptions. To quantify the fidelity of captions, we introduce VidCapQA, a caption-then-QA benchmark with 1,000 questions across 14 capability dimensions. Results on VidCapQA show that captions directly generated by strong VLMs still miss many visual details, highlighting caption representation as a critical bottleneck. Experiments show that CodecCap significantly surpasses direct captioning with the same underlying VLMs, suggesting keyframe-residual captioning a way for high-fidelity video-language supervision. We further use CodecCap to construct CodecVDC-100K, a large-scale dense captioning dataset with anchor, residual, scene-level, and video-level supervision.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。