用双变压器模型提升多机位剪辑选镜准确率,效果远超现有方法。
A Dual-Transformer for Multi-Camera View Recommendation

- 分两阶段处理:先建历史帧记忆,再让各视角独立查询上下文。
- 在TVMCE数据集上达56.60%精度,比之前最优提升近20个百分点。
- 仅用20%视频微调即可适配特定导演风格,适合个性化内容生产。
多摄像机系统是现代媒体制作的基础,多摄像机编辑是一项关键任务,涉及在每个时刻选择合适的摄像机视角。本文提出一种新型双变压器架构,结合交叉注意力机制,在TV Shows Multicamera Editing(TVMCE)数据集上显著超越当前最优模型。该模型将任务解耦:(1) 专用时序编码器首先处理过去帧序列,构建近期历史的丰富记忆;(2) 候选摄像机视角作为查询,通过交叉注意力模块向该记忆提问,使每个候选能独立检索相关上下文信息以进行评估。该方法在测试中达到56.60% [email protected],显著优于此前最佳结果37.16%。进一步消融实验表明,SwinV2骨干网络表现最优,达到69.65% [email protected]。在此最优配置下,我们研究了模型适配特定人类编辑风格的可行性,通过使用目标视频初始片段的不同比例进行微调。结果表明,即使仅使用20%的视频微调,模型在[email protected]上仍表现出可测量提升,显示出强数据效率,具备为每部电视剧或制片人定制编辑风格的巨大潜力。
原文摘要 · Abstract (English)
Multi-camera systems are foundational to modern media production, and multi-camera editing is a critical task. This involves the proper selection of the appropriate camera view at each moment. In this paper, we propose a novel Dual-Transformer architecture with Cross-Attention that heavily outperformed the current SOTA models over the TVMCE dataset (TV Shows Multicamera Editing dataset). Our model decouples these tasks: (1) a dedicated temporal encoder first processes the sequence of past frames to build a rich memory of the recent history, and (2) the candidate camera views then act as queries to this memory via a cross-attention module, allowing each candidate to independently interrogate the historical context and find the most relevant information for its own evaluation. Our approach achieved 56.60% [email protected], representing a substantial improvement over the prior best result of 37.16%. We further conducted an ablation study exploring the use of lightweight backbone architectures, where the SwinV2 backbone yielded the best performance, achieving 69.65% [email protected]. Using this best-performing configuration, we then investigated the feasibility of adapting the model to replicate the editing style of a specific human editor. To this end, we fine-tuned the model using varying proportions of the initial segment of a target video. Our results demonstrate that even with only 20% of the video used for fine-tuning, the model exhibited measurable improvements in [email protected], indicating strong potential for data-efficient personalization of editing style adapted to each individual TV show or producer.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。