用原型学习对齐多视角或AI生成视频，提升同步精度与效率
Synchronization of Multiple Videos
- 基于预训练模型提取嵌入，构建共享1D原型序列作为动作阶段锚点
- 在多个数据集上实现更高同步准确率，相比传统方法减少配对计算量
- 首次有效解决多组生成式AI视频的同步问题，适合跨场景视频分析
从同一场景中多摄像头同步拍摄的视频通常只需简单时间偏移即可对齐。然而，来自不同场景或近期出现的生成式AI视频，由于主体、背景差异及非线性时间错位，同步难度显著增加。本文提出时序原型学习（TPL），一种基于原型的框架，可从任意预训练模型提取的高维嵌入中构建共享紧凑的一维表示。通过学习统一的原型序列来锚定关键动作阶段，避免繁琐的两两匹配。实验表明，TPL在多种数据集上提升了同步准确性、效率和鲁棒性，涵盖细粒度帧检索与相位分类任务。尤为重要的是，TPL是首个有效缓解多组生成式AI视频同步问题的方法。代码与新构建的多视频同步数据集已公开于 https://bgu-cs-vil.github.io/TPL/
原文摘要 · Abstract (English)
Synchronizing videos captured simultaneously from multiple cameras in the same scene is often easy and typically requires only simple time shifts. However, synchronizing videos from different scenes or, more recently, generative AI videos, poses a far more complex challenge due to diverse subjects, backgrounds, and nonlinear temporal misalignment. We propose Temporal Prototype Learning (TPL), a prototype-based framework that constructs a shared, compact 1D representation from high-dimensional embeddings extracted by any of various pretrained models. TPL robustly aligns videos by learning a unified prototype sequence that anchors key action phases, thereby avoiding exhaustive pairwise matching. Our experiments show that TPL improves synchronization accuracy, efficiency, and robustness across diverse datasets, including fine-grained frame retrieval and phase classification tasks. Importantly, TPL is the first approach to mitigate synchronization issues in multiple generative AI videos depicting the same action. Our code and a new multiple video synchronization dataset are available at https://bgu-cs-vil.github.io/TPL/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。