用多尺度向量量化实现快速3D物体生成与理解。
SAR3D: Autoregressive 3D Object Generation and Understanding via Multi-scale 3D VQVAE
- 通过多尺度3D VQVAE将物体分层编码，逐级预测
- 0.82秒完成3D生成，速度远超现有方法
- 可让大模型理解3D内容并自动生成描述
自回归模型在大语言模型、多模态模型和2D内容生成中已取得显著进展，正逐步接近通用人工智能（AGI）。然而，将其应用于3D物体生成与理解仍处于探索阶段。本文提出规模自回归3D（SAR3D）框架，利用多尺度3D向量量化变分自编码器（VQVAE）对3D物体进行分层编码，实现高效自回归生成与深度理解。该方法不直接预测单个令牌，而是逐级预测多尺度潜在表示，大幅降低生成时间，在A6000 GPU上仅需0.82秒即可完成3D生成。同时，基于蕴含层次化3D信息的令牌，我们在预训练大语言模型（LLM）上进行微调，使模型具备对3D内容的多模态理解能力。实验表明，SAR3D在生成速度与质量上均超越现有3D生成方法，并能全面解析与描述3D模型。
原文摘要 · Abstract (English)
Autoregressive models have demonstrated remarkable success across various fields, from large language models (LLMs) to large multimodal models (LMMs) and 2D content generation, moving closer to artificial general intelligence (AGI). Despite these advances, applying autoregressive approaches to 3D object generation and understanding remains largely unexplored. This paper introduces Scale AutoRegressive 3D (SAR3D), a novel framework that leverages a multi-scale 3D vector-quantized variational autoencoder (VQVAE) to tokenize 3D objects for efficient autoregressive generation and detailed understanding. By predicting the next scale in a multi-scale latent representation instead of the next single token, SAR3D reduces generation time significantly, achieving fast 3D object generation in just 0.82 seconds on an A6000 GPU. Additionally, given the tokens enriched with hierarchical 3D-aware information, we finetune a pretrained LLM on them, enabling multimodal comprehension of 3D content. Our experiments show that SAR3D surpasses current 3D generation methods in both speed and quality and allows LLMs to interpret and caption 3D models comprehensively.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。