让视频描述过程可追溯,提升理解准确性。
PercepCap: Video Captioner with Structured Spatio-Temporal Perception

- 先生成物体轨迹和事件时间线,再基于感知证据写描述。
- 在多个数据集上超越基线模型,提升描述准确率。
- 适合需要可解释性视频理解的研究与应用。
视频描述需要精细的时空理解,包括物体位置的空间感知和事件发生的时间感知。现有多模态大模型通常直接从视频输入生成描述,未暴露描述背后的感知依据,导致时空感知错误只能在最终描述中体现,难以定位。为此,我们提出 PercepCap,一个感知可见的视频描述框架,将感知证据显式化后再生成最终描述。具体地,PercepCap 采用‘感知-描述’生成链:模型先生成包含物体轨迹和时间事件的时空感知痕迹,再基于此生成最终描述。为支持该设计,我们提出两阶段训练策略:先通过‘感知-描述’监督微调,使模型适应新生成链;再通过感知引导的强化学习,联合优化感知链与描述质量。为此,我们构建了标注对齐的感知数据:先生成仅含描述的文本,提取其中提及的对象与事件,再回溯至视频中定位框与时间戳,形成与描述对齐的感知数据。该方法在直接描述与描述转问答评估中均优于 Qwen3-VL 基线,达到领先水平。
原文摘要 · Abstract (English)
Video captioning requires fine-grained spatio-temporal understanding of videos, including spatial perception of where objects are located and temporal perception of when events occur. Existing MLLMs usually generate captions directly from video inputs without exposing the perceptual evidence behind descriptions. As a result, mistakes in spatiotemporal perception are only observed in the final caption, making it difficult to identify the underlying perceptual errors directly. To address these issues, we present PercepCap, a perception-aware video captioning framework that makes perceptual evidence explicit before producing the final caption. Specifically, PercepCap follows a perceive-describe generation chain, where the model first produces a spatiotemporal perception trace comprising object trajectories and temporal events, and then generates the final caption conditioned on the perceived evidence. To support this, we design a two-stage training strategy. Perceive-then-Describe Supervised Fine-tuning adapts the model from caption-only generation to the proposed perceive-describe chain, while Perception-Grounded Reinforcement Learning optimizes perception trace and caption quality with joint rewards over perception chain and the final caption. To support our two-stage training, we introduce Caption-Anchored Perception Data Construction. This pipeline builds the SFT and RL training data by first generating a caption-only description, extracting the objects and events it mentions, and grounding them back in the video with boxes and timestamps. This yields caption-aligned perception data that provides solid training ground truth, ensuring that the explicit perception trace and final caption refer to the same objects and events. Across direct caption and caption-to-QA evaluation, PercepCap consistently improves upon the Qwen3-VL baseline and demonstrates leading caption quality.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。