arXiv:2412.02202cs.CV2024-12被引 10

用变分分词器实现3D生成的极致压缩,仅用512字节即可还原高精度3D形状。

3D representation in 512-Byte:Variational tokenizer is the key for autoregressive 3D generation

  • 提出变分分词器(VAT),将无序3D数据转为带隐式层级的紧凑离散标记。
  • 实现最高2000倍压缩率,256个int8标记仍保持92%的形状匹配度。
  • 适合追求高效3D生成与极小模型体积的开发者和研究者。

自回归Transformer已革新高保真图像生成。关键在于分词器,它将高分辨率图像块压缩为适合大语言模型的有序离散标记。然而,将此类分词器扩展至3D生成面临重大挑战:与具有自然空间序列和多尺度关系的图像块不同,3D数据缺乏固有顺序,难以在保留结构细节的同时压缩为少量标记。为此,我们提出变分分词器(VAT),将无序3D数据转化为具有隐式层级的紧凑潜在标记,适用于高效的粗到精自回归建模。VAT首先使用上下文感知Transformer,将大量无序3D特征压缩为信息损失最小的精简标记集。该潜在空间随后映射到高斯分布,通过残差量化逐步增加标记数量。不同尺度的标记在同一大类高斯分布中自动分配至不同子空间,天然建立跨尺度的离散关系,便于建模。解码阶段,利用高分辨率三平面将这些紧凑潜在标记转换为详细3D形状。大量实验表明,VAT实现了可扩展、高效的3D生成,在质量、效率和泛化性上优于现有方法。惊人的是,其压缩率高达250倍,1MB网格可缩减至3.9KB,F-score达96%;进一步压缩至256个int8标记时,仍保持2000倍压缩率和92% F-score。

原文摘要 · Abstract (English)

Autoregressive transformers have revolutionized high-fidelity image generation. One crucial ingredient lies in the tokenizer, which compresses high-resolution image patches into manageable discrete tokens with a scanning or hierarchical order suitable for large language models. Extending these tokenizers to 3D generation, however, presents a significant challenge: unlike image patches that naturally exhibit spatial sequence and multi-scale relationships, 3D data lacks an inherent order, making it difficult to compress into fewer tokens while preserving structural details. To address this, we introduce the Variational Tokenizer (VAT), which transforms unordered 3D data into compact latent tokens with an implicit hierarchy, suited for efficient and high-fidelity coarse-to-fine autoregressive modeling. VAT begins with an in-context transformer, which compress numerous unordered 3D features into a reduced token set with minimal information loss. This latent space is then mapped to a Gaussian distribution for residual quantization, with token counts progressively increasing across scales. In this way, tokens at different scales naturally establish the interconnections by allocating themselves into different subspaces within the same Gaussian distribution, facilitating discrete modeling of token relationships across scales. During the decoding phase, a high-resolution triplane is utilized to convert these compact latent tokens into detailed 3D shapes. Extensive experiments demonstrate that VAT enables scalable and efficient 3D generation, outperforming existing methods in quality, efficiency, and generalization. Remarkably, VAT achieves up to a 250x compression, reducing a 1MB mesh to just 3.9KB with a 96% F-score, and can further compress to 256 int8 tokens, achieving a 2000x reduction while maintaining a 92% F-score.

3D生成变分分词压缩自回归

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。