用自适应纠错机制生成电影预告片,效果优于现有方法。
Self-Paced and Self-Corrective Masked Prediction for Movie Trailer Generation
- 通过双向建模与渐进式自修正预测生成预告片
- 在多个数据集上超越现有方法,生成质量更优
- 适合视频编辑、内容生成领域的研究者和开发者
电影预告片生成是一项具有挑战性的视频编辑任务,涉及从影片中选择并重组镜头以制作吸引人的预告片。当前大多数自动预告片生成方法采用“选择-排序”范式(即先选择关键镜头再排序),存在误差传播问题,限制了生成质量。为此,本文提出一种新的自适应节奏与自校正掩码预测方法SSMP,通过双向上下文建模与渐进式自我修正,实现自动预告片生成的最先进性能。SSMP训练一个Transformer编码器,以影片镜头序列为提示,生成对应的预告片镜头序列。模型通过掩码预测进行训练,从随机掩码的预告片序列中重建完整序列。掩码比例自适应调整,使任务难度随模型能力动态变化,提升性能。生成时,模型逐步填充高置信度镜头位置,并对剩余位置重新掩码,形成类似人类编辑的渐进式自修正过程。定量结果与用户研究表明,SSMP显著优于现有方法。演示代码已开源:https://github.com/Dixin-Lab/SSMP。
原文摘要 · Abstract (English)
As a challenging video editing task, movie trailer generation involves selecting and reorganizing movie shots to create engaging trailers. Currently, most existing automatic trailer generation methods employ a "selection-then-ranking" paradigm (i.e., first selecting key shots and then ranking them), which suffers from inevitable error propagation and limits the quality of the generated trailers. Beyond this paradigm, we propose a new self-paced and self-corrective masked prediction method called SSMP, which achieves state-of-the-art results in automatic trailer generation via bi-directional contextual modeling and progressive self-correction. In particular, SSMP trains a Transformer encoder that takes the movie shot sequences as prompts and generates corresponding trailer shot sequences accordingly. The model is trained via masked prediction, reconstructing each trailer shot sequence from its randomly masked counterpart. The mask ratio is self-paced, allowing the task difficulty to adapt to the model and thereby improving model performance. When generating a movie trailer, the model fills the shot positions with high confidence at each step and re-masks the remaining positions for the next prediction, forming a progressive self-correction mechanism that is analogous to how human editors work. Both quantitative results and user studies demonstrate the superiority of SSMP in comparison to existing automatic movie trailer generation methods. Demo is available at: https://github.com/Dixin-Lab/SSMP.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。