用视频生成模型统一修复多种3D渲染的伪影,无需重训练。
FixAnything: 3D-Consistent Rendering Refinement via Video Generative Priors

- 复用预训练视频模型,通过视频到视频转换修复渲染缺陷。
- 在4种3D表示上均提升质量,仅需轻量微调,效果稳定。
- 适合需要快速部署高质量渲染修复的开发者和研究者。
使用3D场景表示(如高斯溅射、神经辐射场、网格或点云)渲染视图时,若输入视图稀疏或目标视图远离输入,会产生伪影。现有方法虽利用基于扩散的生成先验缓解问题,但针对特定表示,需定制架构或大量重训练。本文提出FixAnything,一个可修复多种渲染伪影的通用模型。其核心思想是:即使噪声较大的序列也保留相机运动和粗粒度场景结构,因此可将修复建模为视频到视频翻译。通过引入二值掩码标记干净像素,模型能锚定输出至高质量输入(如训练视图),同时优化其余部分。为确保输出3D一致性并支持下游重建,采用结构从运动恢复的相机位姿精度作为直接偏好优化(DPO)的奖励信号。在四种不同3D表示上,FixAnything均以轻量微调显著提升渲染质量,证明单一通用视频先验可替代多个专用修复流水线。该框架简洁,未来可无缝接入更强视频模型,无需重新设计架构。
原文摘要 · Abstract (English)
Rendering views using 3D scene representations such as Gaussian Splatting (3DGS), Neural Radiance Fields (NeRF), meshes, or even point clouds produces artifacts when input views are sparse or target views lie far from the input. Recent work mitigates these artifacts using diffusion-based generative priors, but is specialized to individual representations and require custom architectures or extensive retraining. We present FixAnything, a single model for fixing a wide range of rendering artifacts. It does so by repurposing a pretrained video generative model, leveraging its implicit multi-view priors with only minimal modification and lightweight finetuning. Our key insight is that even noisily-rendered sequences preserve camera motion and coarse scene structure, allowing cleanup to be formulated as video-to-video translation. To control what scene structure should be preserved, we introduce a binary mask denoting the clean pixels, enabling the model to anchor its output to high-quality inputs (e.g. training views) while refining the rest. To encourage FixAnything to produce 3D-consistent renderings that support downstream reconstruction, we use camera pose accuracy (recovered via structure-from-motion) as a reward signal for direct preference optimization (DPO). Across four distinct 3D representations, FixAnything consistently improves rendering quality with lightweight finetuning, demonstrating that a single generalist video prior can replace multiple specialist refinement pipelines. The simplicity of the framework enables immediate adoption of stronger future video models without architectural redesign.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。