用4D高斯光追技术一次生成带真实镜头效果的动态视频数据
Every Camera Effect, Every Time, All at Once: 4D Gaussian Ray Tracing for Physics-based Camera Effect Data Generation
- 先用4D高斯点云重建动态场景，再用物理光追模拟镜头效果
- 生成速度最快，画质优于或媲美现有方法
- 构建了8个室内动态场景基准，支持四种镜头效果评估
主流计算机视觉系统通常假设理想针孔相机，但在面对鱼眼畸变、果冻效应等真实镜头效果时表现不佳，主要因缺乏带有镜头效果的训练数据。现有数据生成方法要么成本高，要么存在仿真到现实的差距，或无法准确建模镜头效果。为此，我们提出4D高斯光追（4D-GRT），一种结合4D高斯喷溅与物理光追的两阶段新方法。给定多视角视频，4D-GRT首先重建动态场景，再通过光追生成可控制、物理精确的镜头效果视频。该方法在渲染速度上最快，同时在画质上优于或媲美现有基线。此外，我们构建了八个室内环境下的合成动态场景，涵盖四种镜头效果，作为评估镜头效果生成视频的基准。
原文摘要 · Abstract (English)
Common computer vision systems typically assume ideal pinhole cameras but fail when facing real-world camera effects such as fisheye distortion and rolling shutter, mainly due to the lack of learning from training data with camera effects. Existing data generation approaches suffer from either high costs, sim-to-real gaps or fail to accurately model camera effects. To address this bottleneck, we propose 4D Gaussian Ray Tracing (4D-GRT), a novel two-stage pipeline that combines 4D Gaussian Splatting with physically-based ray tracing for camera effect simulation. Given multi-view videos, 4D-GRT first reconstructs dynamic scenes, then applies ray tracing to generate videos with controllable, physically accurate camera effects. 4D-GRT achieves the fastest rendering speed while performing better or comparable rendering quality compared to existing baselines. Additionally, we construct eight synthetic dynamic scenes in indoor environments across four camera effects as a benchmark to evaluate generated videos with camera effects.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。