用自回归模型实现单图视角合成，生成更符合相机轨迹顺序。
ARSS: Taming Decoder-only Autoregressive Visual Generation for View Synthesis From Single View
- 采用GPT风格解码器，按相机轨迹逐帧生成图像。
- 在MVD和DTU数据集上达到与扩散模型相当的视觉质量。
- 创新性地打乱空间顺序提升细节，同时保持时间因果性。
扩散模型在世界建模任务中表现优异，包括从稀疏输入生成新视角。但现有基于扩散的视角合成方法通过迭代去噪联合生成目标视角，难以严格遵循相机轨迹的因果结构。相比之下，自回归（AR）模型以因果方式生成，每个图像块仅依赖先前生成的内容。本文提出ARSS框架，使用GPT风格的解码器仅凭单张图像，根据预定义相机轨迹生成新视角。采用现成视频分词器将连续图像序列转为离散标记，并设计相机编码器将轨迹转换为3D位置引导信号。为进一步提升生成质量且不破坏自回归结构，提出一种自回归变换模块，在保持时间顺序的同时随机打乱空间标记顺序。在公开数据集上的定性和定量实验表明，本方法性能接近当前最优扩散模型方案。
原文摘要 · Abstract (English)
Diffusion models have achieved impressive results in world modeling tasks, including novel view generation from sparse inputs. However, most existing diffusion-based NVS methods generate target views jointly via an iterative denoising process, which makes it less straightforward to impose a strictly causal structure along a camera trajectory. In contrast, autoregressive (AR) models operate in a causal fashion, generating each token based on all previously generated tokens. In this work, we introduce ARSS, a novel framework that leverages a GPT-style decoder-only AR model to generate novel views from a single image, conditioned on a predefined camera trajectory. We employ an off-the-shelf video tokenizer to map continuous image sequences into discrete tokens and propose a camera encoder that converts camera trajectories into 3D positional guidance. Then to enhance generation quality while preserving the autoregressive structure, we propose an autoregressive transformer module that randomly permutes the spatial order of tokens while maintaining their temporal order. Qualitative and quantitative experiments on public datasets demonstrate that our method achieves overall performance comparable to state-of-the-art view synthesis approaches based on diffusion models. Project page: https://wbteng9526.github.io/arss/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。