用自回归Transformer生成3D高斯场景，支持可控、渐进式构建。
GaussianGPT: Towards Autoregressive 3D Gaussian Scene Generation
- 通过稀疏3D卷积自编码器+向量量化压缩高斯基元为离散令牌，再用因果Transformer逐个生成。
- 实现3D场景的逐步生成，支持补全、外扩和温度控制采样，生成过程灵活可控。
- 适合需要精细控制、分步生成的3D内容创作，如数字孪生、游戏场景设计。
近期3D生成模型多基于扩散或流匹配范式。本文提出GaussianGPT，一种基于Transformer的全自回归模型，直接通过预测下一个令牌生成3D高斯，实现完整3D场景生成。首先使用带有向量量化（VQ）的稀疏3D卷积自编码器将高斯基元压缩为离散潜在网格，随后对生成的令牌序列化并采用带3D旋转位置编码的因果Transformer建模，实现空间结构与外观的逐步生成。与扩散方法整体优化不同，该方法分步构建场景，天然支持补全、外扩、温度控制采样及灵活生成长度。该范式利用自回归模型的组合归纳偏置与可扩展性，同时在显式表示上兼容现代神经渲染管线，使自回归Transformer成为可控、上下文感知3D生成的互补范式。
原文摘要 · Abstract (English)
Most recent advances in 3D generative modeling rely on diffusion or flow-matching formulations. We instead explore a fully autoregressive alternative and introduce GaussianGPT, a transformer-based model that directly generates 3D Gaussians via next-token prediction, thus facilitating full 3D scene generation. We first compress Gaussian primitives into a discrete latent grid using a sparse 3D convolutional autoencoder with vector quantization. The resulting tokens are serialized and modeled using a causal transformer with 3D rotary positional embedding, enabling sequential generation of spatial structure and appearance. Unlike diffusion-based methods that refine scenes holistically, our formulation constructs scenes step-by-step, naturally supporting completion, outpainting, controllable sampling via temperature, and flexible generation horizons. This formulation leverages the compositional inductive biases and scalability of autoregressive modeling while operating on explicit representations compatible with modern neural rendering pipelines, positioning autoregressive transformers as a complementary paradigm for controllable and context-aware 3D generation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。