无需预训练2D模型，仅用3D渲染数据即可实现高质量单图视角生成。
Next-Scale Autoregressive Models are Zero-Shot Single-Image Object View Synthesizers
- 基于下一代自回归框架，结合全局语义与多尺度局部编码进行视图条件控制。
- 在无2D预训练情况下，性能超越现有方法，且推理速度提升数倍。
- 适用于复杂视角生成，适合追求高效高质图像合成的研究者。
基于扩散模型的新型视图合成（NVS）方法虽已取得突破，但依赖昂贵的2D预训练模型（如Stable Diffusion）作为几何先验，严重限制了其可扩展性。本文提出ArchonView，一种完全从零训练的下一代自回归视图合成方法，仅使用3D渲染数据，无需2D预训练。通过引入姿态增强语义的全局条件和多尺度层级编码的局部条件，模型在多个任务中表现优于现有方法。即使在以往方法失效的困难相机视角下，仍保持稳健性能，推理速度较扩散模型快数倍。实验验证性能随模型与数据规模增长，且代码已开源。
原文摘要 · Abstract (English)
Methods based on diffusion backbones have recently revolutionized novel view synthesis (NVS). However, those models require pretrained 2D diffusion checkpoints (e.g., Stable Diffusion) as the basis for geometrical priors. Since such checkpoints require exorbitant amounts of data and compute to train, this greatly limits the scalability of diffusion-based NVS models. We present Next-Scale Autoregression Conditioned by View (ArchonView), a method that significantly exceeds state-of-the-art methods despite being trained from scratch with 3D rendering data only and no 2D pretraining. We achieve this by incorporating both global (pose-augmented semantics) and local (multi-scale hierarchical encodings) conditioning into a backbone based on the next-scale autoregression paradigm. Our model also exhibits robust performance even for difficult camera poses where previous methods fail, and is several times faster in inference speed compared to diffusion. We experimentally verify that performance scales with model and dataset size, and conduct extensive demonstration of our method's synthesis quality across several tasks. Our code is open-sourced at https://github.com/Shiran-Yuan/ArchonView.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。