让模型理解变化过程,生成更准确的动态描述。
Imagine How To Change: Explicit Procedure Modeling for Change Captioning
- 通过关键帧建模变化过程,显式捕捉动态演化。
- 在三个数据集上显著优于现有方法,提升描述准确性。
- 适合需要理解变化细节的任务,如视频分析、自动驾驶。
变化描述任务旨在生成明确描述两张视觉相似图像差异的文本。现有方法基于静态图像对,忽略了变化过程中的丰富时间动态,而这一动态正是理解‘变化了什么’和‘如何变化’的关键。本文提出 ProCap 框架,将变化建模从静态对比转向动态过程建模。第一阶段通过自动生成中间帧,显式化隐含的程序性动态,并采样关键帧以减少冗余;再利用基于图文提示的掩码重建任务,训练一个过程编码器来捕捉这些关键帧的潜在动态。第二阶段将训练好的编码器集成到编码器-解码器结构中进行描述生成。为避免依赖前阶段显式帧(带来计算开销和对噪声敏感),引入可学习的过程查询,驱动编码器推断潜在过程表示,由解码器转化为文本。整个模型端到端训练,确保输出兼具时间连贯性和与描述的一致性。在三个数据集上的实验验证了 ProCap 的有效性。代码与预训练模型已开源。
原文摘要 · Abstract (English)
Change captioning generates descriptions that explicitly describe the differences between two visually similar images. Existing methods operate on static image pairs, thus ignoring the rich temporal dynamics of the change procedure, which is the key to understand not only what has changed but also how it occurs. We introduce ProCap, a novel framework that reformulates change modeling from static image comparison to dynamic procedure modeling. ProCap features a two-stage design: The first stage trains a procedure encoder to learn the change procedure from a sparse set of keyframes. These keyframes are obtained by automatically generating intermediate frames to make the implicit procedural dynamics explicit and then sampling them to mitigate redundancy. Then the encoder learns to capture the latent dynamics of these keyframes via a caption-conditioned, masked reconstruction task. The second stage integrates this trained encoder within an encoder-decoder model for captioning. Instead of relying on explicit frames from the previous stage -- a process incurring computational overhead and sensitivity to visual noise -- we introduce learnable procedure queries to prompt the encoder for inferring the latent procedure representation, which the decoder then translates into text. The entire model is then trained end-to-end with a captioning loss, ensuring the encoder's output is both temporally coherent and captioning-aligned. Experiments on three datasets demonstrate the effectiveness of ProCap. Code and pre-trained models are available at https://github.com/BlueberryOreo/ProCap
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。