用3D定位与镜头匹配技术，重建电视剧场景并支持自由编辑镜头和演员。
ShowMak3r: Compositional TV Show Reconstruction
- 通过深度先验与插值估计未见姿态，实现演员在复杂场景中的3D定位。
- 在Sitcoms3D数据集上实现新视角拍摄与时间点重播，重建准确率显著提升。
- 适合影视制作、虚拟拍摄及动画生成领域，支持换镜头、换演员等操作。
从电视节目视频中重建动态辐射场极具挑战性，尤其面对演员相互遮挡、表情多样、布景杂乱以及视点基线小或突然换镜等问题。为此，我们提出ShowMak3r，一个完整的重建流程，可像在制作控制室中编辑视频一样操作场景。其3DLocator模块利用深度先验定位演员，并通过插值估算未见人体姿态；ShotMatcher模块可在镜头切换下持续追踪演员；此外，引入面部拟合网络动态恢复演员表情。在Sitcoms3D数据集上的实验表明，该流程可实现不同时间戳下新相机视角的场景重装。我们还展示了合成镜头、演员移位、插入、删除及姿态操控等有趣应用。
原文摘要 · Abstract (English)
Reconstructing dynamic radiance fields from video clips is challenging, especially when entertainment videos like TV shows are given. Many challenges make the reconstruction difficult due to (1) actors occluding with each other and having diverse facial expressions, (2) cluttered stages, and (3) small baseline views or sudden shot changes. To address these issues, we present ShowMak3r, a comprehensive reconstruction pipeline that allows the editing of scenes like how video clips are made in a production control room. In ShowMak3r, a 3DLocator module locates recovered actors on the stage using depth prior and estimates unseen human poses via interpolation. The proposed ShotMatcher module then tracks the actors under shot changes. Furthermore, ShowMak3r introduces a face-fitting network that dynamically recovers the actors' expressions. Experiments on Sitcoms3D dataset show that our pipeline can reassemble TV show scenes with new cameras at different timestamps. We also demonstrate that ShowMak3r enables interesting applications such as synthetic shot-making, actor relocation, insertion, deletion, and pose manipulation. Project page : https://nstar1125.github.io/showmak3r
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。