用可微渲染提升视频预测细节,告别模糊画面。
Learning Video Dynamics with Predictive Differentiable Rendering

- 引入2D高斯表示的轻量适配器,无缝融合现有模型
- 结合L1与SSIM损失,实现10倍加速且细节更清晰
- 适合需要高保真视频生成的研究者与工业应用
如何准确预测高保真未来世界?现有确定性视频预测模型在离散像素空间中运行,主要使用像素级均方误差(MSE)优化,常导致预测结果过平滑、缺乏细粒度视觉细节。为解决这一问题,我们提出预测可微渲染(PDR),一种连接离散与连续表示的新范式。受3D高斯点云重建启发,提出PredGS,基于2D高斯表示的轻量级即插即用适配器,可无缝集成至现有像素空间预测器,显著提升空间细节保留,计算开销几乎忽略不计。进一步开发predgsplat,一个支持任意通道的CUDA加速可微2D高斯渲染器,每个高斯包含5 + C个可学习参数(位置、尺度、旋转及C个通道振幅),渲染速度达基线的10倍。通过联合使用L1和SSIM损失优化,PDR克服了MSE损失固有的模糊倾向,显著提升预测性能。在TaxiBJ、WeatherBench、KTH和Human3.6M等多个真实世界基准上的大量实验表明,PDR持续优于现有方法,在细节保留、视觉保真度和预测精度上表现优异。
原文摘要 · Abstract (English)
How to accurately predict a high-fidelity future world? While the visual world is inherently continuous, existing deterministic video prediction models operate in discrete pixel space and are mainly optimized with pixel-wise mean squared error (MSE), which often leads to over-smoothed predictions and a lack of fine-grained visual details. To address these limitations, we propose Predictive Differentiable Rendering (PDR), a novel end-to-end video prediction paradigm that bridges the gap between discrete and continuous representations. Inspired by recent progress in 3D reconstruction with 3D Gaussian Splatting, we introduce PredGS, a lightweight and plug-and-play adapter based on 2D Gaussian representation, which could be seamlessly integrated with existing pixel space predictors, significantly improving spatial detail preservation with negligible computational overhead. Furthermore, we develop predgsplat, a CUDA-accelerated differentiable 2D Gaussian renderer supporting arbitrary channels. Each Gaussian is defined by 5 + C learnable parameters (position, scale, rotation, and C channel amplitudes) and achieves up to 10x faster rendering than the baseline. Optimized by a combined L1 and SSIM loss, PDR overcomes the inherent blurring tendencies of MSE Loss, significantly enhancing the prediction performance. Extensive experiments on diverse real-world benchmarks, including TaxiBJ, WeatherBench, KTH, and Human3.6M, demonstrate that PDR consistently surpasses existing methods, delivering superior detail preservation, visual fidelity, and predictive accuracy.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。