arXiv:2410.00890cs.CVcs.GR2024-10ICML被引 14

Flex3D可任意输入高质量视图,生成更优3D内容。

Flex3D: Feed-Forward 3D Generation with Flexible Reconstruction Model and Input View Curation

  • 用扩散模型生成候选视图,再筛选高质量视图用于重建
  • 采用可处理任意数量输入的Transformer结构,生成高精度3D点云
  • 在多项指标上超越现有方法,用户评测胜率超92%

从文本、单图或稀疏视图生成高质量3D内容仍是挑战性任务。现有方法通常使用多视图扩散模型合成多视角图像,再通过前馈流程进行3D重建,但受限于固定且少量的输入视图,难以捕捉多样视角,若合成视图质量差则导致生成效果不佳。为此,我们提出Flex3D,一种两阶段新框架,可利用任意数量的高质量输入视图。第一阶段包含候选视图生成与筛选:使用微调的多视图图像扩散模型和视频扩散模型生成候选视图池,实现目标3D物体的丰富表征;随后通过视图选择管道基于质量和一致性过滤,确保仅保留高可靠性的视图用于重建。第二阶段将筛选后的视图输入柔性重建模型(FlexRM),该模型基于Transformer架构,能有效处理任意数量输入,直接输出3D Gaussian点云,采用三平面表示,实现高效且细节丰富的3D生成。通过深入探索设计与训练策略,优化了FlexRM在重建与生成任务中的表现。实验表明,Flex3D在多项指标上达到当前最优水平,用户研究中在3D生成任务上的胜率超过92%。

原文摘要 · Abstract (English)

Generating high-quality 3D content from text, single images, or sparse view images remains a challenging task with broad applications. Existing methods typically employ multi-view diffusion models to synthesize multi-view images, followed by a feed-forward process for 3D reconstruction. However, these approaches are often constrained by a small and fixed number of input views, limiting their ability to capture diverse viewpoints and, even worse, leading to suboptimal generation results if the synthesized views are of poor quality. To address these limitations, we propose Flex3D, a novel two-stage framework capable of leveraging an arbitrary number of high-quality input views. The first stage consists of a candidate view generation and curation pipeline. We employ a fine-tuned multi-view image diffusion model and a video diffusion model to generate a pool of candidate views, enabling a rich representation of the target 3D object. Subsequently, a view selection pipeline filters these views based on quality and consistency, ensuring that only the high-quality and reliable views are used for reconstruction. In the second stage, the curated views are fed into a Flexible Reconstruction Model (FlexRM), built upon a transformer architecture that can effectively process an arbitrary number of inputs. FlemRM directly outputs 3D Gaussian points leveraging a tri-plane representation, enabling efficient and detailed 3D generation. Through extensive exploration of design and training strategies, we optimize FlexRM to achieve superior performance in both reconstruction and generation tasks. Our results demonstrate that Flex3D achieves state-of-the-art performance, with a user study winning rate of over 92% in 3D generation tasks when compared to several of the latest feed-forward 3D generative models.

3D生成扩散模型多视图Transformer

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。