arXiv:2409.06322cs.CV2024-09IJCAI被引 15

用跨尺度查询实现3D生成的自回归建模,突破无序点云难题。

G3PT: Unleash the power of Autoregressive Modeling in 3D Generation via Cross-scale Querying Transformer

论文配图:G3PT: Unleash the power of Autoregressive Modeling in 3D Generation via Cross-scale Querying Transformer
图 1 · 摘自论文原文
  • 通过多级离散化点云建立自然序列,适配自回归模型
  • 支持多种条件输入,生成质量超越现有方法
  • 首次发现3D生成中规模扩展的幂律规律,适合研究者参考

自回归变换器在语言和图像生成中已取得显著进展,但在3D生成中因依赖顺序预测而面临挑战,因其难以适配无序的3D数据。本文提出G3PT,一种基于跨尺度查询变换器的可扩展粗到精3D生成模型。核心思想是将基于点的3D数据映射为不同细节层级的离散标记,自然建立多层级间的序列关系,适用于自回归建模。同时,跨尺度查询变换器可在不依赖顺序的前提下,全局连接不同层级的标记。该设计使G3PT具备灵活的生成流程,可轻松支持多种条件结构,实现从各类条件生成3D形状。大量实验表明,相较于先前方法,G3PT在生成质量和泛化能力上均有显著提升。最重要的是,首次在3D生成中观察到模型规模扩展的幂律行为。

原文摘要 · Abstract (English)

Autoregressive transformers have revolutionized generative models in language processing and shown substantial promise in image and video generation. However, these models face significant challenges when extended to 3D generation tasks due to their reliance on next-token prediction to learn token sequences, which is incompatible with the unordered nature of 3D data. Instead of imposing an artificial order on 3D data, in this paper, we introduce G3PT, a scalable coarse-to-fine 3D generative model utilizing a cross-scale querying transformer. The key is to map point-based 3D data into discrete tokens with different levels of detail, naturally establishing a sequential relationship between different levels suitable for autoregressive modeling. Additionally, the cross-scale querying transformer connects tokens globally across different levels of detail without requiring an ordered sequence. Benefiting from this approach, G3PT features a versatile 3D generation pipeline that effortlessly supports diverse conditional structures, enabling the generation of 3D shapes from various types of conditions. Extensive experiments demonstrate that G3PT achieves superior generation quality and generalization ability compared to previous 3D generation methods. Most importantly, for the first time in 3D generation, scaling up G3PT reveals distinct power-law scaling behaviors.

3D生成自回归模型跨尺度建模点云

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。