用自回归预测生成高质量3D模型，支持文本和图像驱动。
TAR3D: Creating High-Quality 3D Assets via Next-Part Prediction
- 将文本/图像转3D任务转化为分步预测代码索引序列。
- 在ShapeNet和Objaverse上生成质量优于现有方法。
- 适合需要高保真3D资产的设计师与游戏开发者。
我们提出TAR3D，一种由3D感知向量量化变分自编码器（VQ-VAE）和生成式预训练变换器（GPT）组成的框架，用于生成高质量3D资产。核心思想是将下一词预测范式迁移至条件3D物体生成任务中。首先，3D VQ-VAE将多种3D形状编码至紧凑的三平面潜在空间，并利用可训练码本的离散表示，在查询点占据监督下重建细粒度几何结构。随后，配备自定义三平面位置嵌入（TriPE）的3D GPT以自回归方式预测码本索引序列，实现几何体逐部分组合。在ShapeNet和Objaverse上的大量实验表明，TAR3D在文本到3D和图像到3D任务中均达到优于现有方法的生成质量。
原文摘要 · Abstract (English)
We present TAR3D, a novel framework that consists of a 3D-aware Vector Quantized-Variational AutoEncoder (VQ-VAE) and a Generative Pre-trained Transformer (GPT) to generate high-quality 3D assets. The core insight of this work is to migrate the multimodal unification and promising learning capabilities of the next-token prediction paradigm to conditional 3D object generation. To achieve this, the 3D VQ-VAE first encodes a wide range of 3D shapes into a compact triplane latent space and utilizes a set of discrete representations from a trainable codebook to reconstruct fine-grained geometries under the supervision of query point occupancy. Then, the 3D GPT, equipped with a custom triplane position embedding called TriPE, predicts the codebook index sequence with prefilling prompt tokens in an autoregressive manner so that the composition of 3D geometries can be modeled part by part. Extensive experiments on ShapeNet and Objaverse demonstrate that TAR3D can achieve superior generation quality over existing methods in text-to-3D and image-to-3D tasks
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。