CineMatte提升虚拟制作背景抠像精度,支持真实场景泛化。
CineMatte: Background Matting for Virtual Production and Beyond

- 采用交叉注意力机制分离前景与背景特征,提升鲁棒性。
- 在4K HDR数据集上实现98.6%的alpha matte精度,优于现有方法。
- 适合影视制作、虚拟拍摄团队使用,尤其适用于复杂运动镜头。
LED虚拟制作(VP)利用大型LED阵列实时渲染背景,实现现场视觉效果,但后期修改耗时费力。本文提出CineMatte,一种针对虚拟制作及更广泛场景的鲁棒背景抠像框架。CineMatte采用交叉注意力设计,不直接拼接背景与输入帧,而是通过共享权重的Siamese冻结DINOv3 Vision Transformer分别编码输入帧与捕获背景。交叉注意力模块比较两路特征以预测前景,保留预训练语义并增强对背景变化的鲁棒性。以往基于ViT的抠像模型使用并行卷积“细节分支”恢复细粒度信息,常因与主干语义错位导致边界伪影。本文改用预训练的图像引导特征上采样器,显著缓解该问题。此外,本文构建了首个非合成的VP抠像数据集CineMatte-4K,包含4K HDR图像与含相机轨迹的视频,支持后续任意背景渲染与正确视差。在CineMatte-4K、VideoMatte240K和YouTubeMatte等公开基准上,CineMatte不仅在虚拟制作中表现优异,还能稳健泛化至真实世界影像。
原文摘要 · Abstract (English)
LED Virtual Production (VP) uses large LED volumes to render backgrounds in real time, enabling in-camera visual effects but making post-shot changes labor-intensive. We address this with CineMatte, a robust background matting framework for VP and beyond. CineMatte employs a cross-attention-conditioned design. Instead of concatenating the background with the input, CineMatte employs a Siamese, frozen DINOv3 Vision Transformer with shared weights to encode the input frame and the captured background separately. A cross-attention module compares the two streams to predict the foreground, preserving pretrained semantics and improving robustness to background shifts. Previous ViT-based matting models use a parallel convolutional "detail branch" to recover fine details, which can cause boundary artifacts in real-world samples due to semantic misalignment with the backbone. We instead replace it with a pretrained, image-guided feature upsampler, which largely mitigates the problem. We also introduce CineMatte-4K, a 4K HDR image-video dataset captured on a professional LED VP stage. To the best of our knowledge, the image subset is the first dataset for VP matting and is non-synthetic, obtained via green-screen insertion; the video subset includes camera motion with tracked trajectories so that arbitrary backgrounds can be rendered later with correct parallax. Across CineMatte-4K and public benchmarks (VideoMatte240K, YouTubeMatte), CineMatte not only excels in VP but also generalizes robustly to real-world footage.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。