针对拍摄舞台视频的抠像难题,提出适配方案与评估方法。
Capture Stage Matting: Challenges, Approaches, and Solutions for Offline and Real-Time Processing
- 梳理拍摄舞台内容的独特挑战,构建问题清单
- 设计无需大量标注的高效处理流水线,支持离线与实时
- 用扩散模型生成验证数据,客观评估算法效果
拍摄舞台是电影、游戏等媒体领域高精度制作的核心来源。几乎所有流程的关键步骤都是抠像,即将表演内容从背景中分离。尽管通用抠像算法在视频会议和移动娱乐中表现优异,但在拍摄舞台内容上仍面临显著挑战。本文系统梳理了此类内容的独特属性,提出针对性干预策略,并为从业者提供优化工作流程的指导。我们还展示了一套高效的处理管道,可将先进算法快速适配至特定拍摄场景,无需大量标注,支持离线与实时处理。为客观评估效果,引入基于先进扩散模型的验证方法,验证所提方案的有效性。
原文摘要 · Abstract (English)
Capture stages are high-end sources of state-of-the-art recordings for downstream applications in movies, games, and other media. One crucial step in almost all pipelines is matting, i.e., separating captured performances from the background. While common matting algorithms deliver remarkable performance in other applications like teleconferencing and mobile entertainment, we found that they struggle significantly with the peculiarities of capture stage content. The goal of our work is to share insights into those challenges as a curated list of these characteristics along with a constructive discussion for proactive intervention and present a guideline to practitioners for an improved workflow to mitigate unresolved challenges. To this end, we also demonstrate an efficient pipeline to adapt state-of-the-art approaches to such custom setups without the need for extensive annotations, both offline and real-time. For an objective evaluation, we introduce a validation methodology using a state-of-the-art diffusion model to demonstrate the benefits of our approach.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。