通过多视角梯度聚合,用单个2D扩散模型生成更一致的3D内容。
Variance Reduction on the Camera Axis: Multi-View Score Distillation for 3D

- 每步采样时融合多个视角梯度,降低噪声并提升形状一致性。
- 在10,000次UNet调用预算下,K=4使CLIP分数达0.307,准确率86.9%。
- 无需重训练或多视图数据,兼容现有得分蒸馏流程,适合3D生成研究者。
得分蒸馏将预训练2D扩散模型转化为3D生成器,但每步梯度仅基于单一随机视角估算,存在高方差且忽视全局形状一致性问题。以往方法通过在多视角数据上重训扩散先验来改善,但这使采样性能与先验质量耦合。本文提出隔离采样轴的思路:每步梯度是视角期望的一个噪声样本,通过固定总UNet预算下聚合K个视角梯度(梯度累积),可降方差而不改动先验。提出多视角聚合得分蒸馏(MV-SDI),以抗对偶视角对的方式选取视点,实现均衡角度覆盖。在10,000次UNet调用预算下,当K=2时,CLIP R-Precision从74.8%提升至83.8%,CLIP分数从0.297升至0.312,在HPSv2和ImageReward上均有提升,43提示基准上发散率为0%;优化步数减半。K=4时步骤减少四倍,R-Precision达86.9%,CLIP为0.307,所有对齐指标均优于单视角基线。该方法兼容基于梯度的得分蒸馏流程(如通过反演的得分蒸馏),无需重训练或多视图数据。
原文摘要 · Abstract (English)
Score distillation turns a pretrained 2D diffusion model into a 3D generator, but the per-step gradient is estimated from a single randomly chosen view: it is high-variance and blind to global shape consistency. Prior work addresses this by retraining the diffusion prior on multi-view data; this improves consistency but makes the sampling contribution inseparable from prior quality. We instead isolate the sampling axis. The per-step gradient is one noisy sample of an expectation over views; aggregating K samples per step at a fixed total UNet budget reduces variance without touching the prior. We introduce Multi-View Aggregated Score Distillation (MV-SDI), which aggregates gradients from K views per step via gradient accumulation, keeping peak memory unchanged and the 2D prior frozen, and draws views as antithetic antipodal pairs, a prior-independent geometric property, for balanced angular coverage. At a fixed 10,000-UNet-call budget, K=2 raises CLIP R-Precision from 74.8% to 83.8% and CLIP score from 0.297 to 0.312, with consistent gains on HPSv2 and ImageReward and a 0.0% divergence rate on the 43-prompt benchmark; optimization steps halve as a consequence. K=4 gives a fourfold step reduction at R-Precision 86.9% and CLIP 0.307, still well above the single-view baseline on every alignment metric. MV-SDI is compatible with gradient-based score-distillation pipelines, including Score Distillation via Inversion, and requires no retraining and no multi-view data.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。