用视频生成模型+3D重建模型拼接,实现文本生成高质量3D场景。
Text-to-3D by Stitching a Multi-view Reconstruction Network to a Video Generator
- 将文本到视频生成器与3D重建网络通过潜在空间层对接。
- 相比之前基于高斯泼溅的方法,3D生成质量显著提升。
- 适合想做高质量文本生成3D或点云的开发者和研究者。
大规模预训练模型在视觉内容生成和3D重建领域的快速发展,为文本到3D生成带来了新可能。直观上,若能结合现代潜空间文本到视频模型(作为生成器)与最新前馈式3D重建系统(作为解码器),即可构建强大的3D场景生成框架。本文提出VIST3A,解决两大挑战:其一,需以保留权重中丰富知识的方式连接两部分。我们重新审视模型拼接,识别出与文本到视频生成器输出潜空间最匹配的3D解码器层,并进行拼接,仅需少量无标签数据;其二,需对齐文本到视频生成器与拼接后的3D解码器,确保生成潜空间可解码为一致且逼真的3D几何结构。为此,我们采用直接奖励微调技术,用于对齐人类偏好。我们在不同视频生成器与3D重建模型组合下评估了VIST3A,所有配对均显著优于先前基于高斯泼溅的文本到3D模型。此外,通过选择合适的3D基础模型,VIST3A还可实现高质量文本到点云生成。
原文摘要 · Abstract (English)
The rapid progress of large, pretrained models for both visual content generation and 3D reconstruction opens up new possibilities for text-to-3D generation. Intuitively, one could obtain a formidable 3D scene generator if one were able to combine the power of a modern latent text-to-video model as "generator" with the geometric abilities of a recent (feedforward) 3D reconstruction system as "decoder". We introduce VIST3A, a general framework that does just that, addressing two main challenges. First, the two components must be joined in a way that preserves the rich knowledge encoded in their weights. We revisit model stitching, i.e., we identify the layer in the 3D decoder that best matches the latent representation produced by the text-to-video generator and stitch the two parts together. That operation requires only a small dataset and no labels. Second, the text-to-video generator must be aligned with the stitched 3D decoder, to ensure that the generated latents are decodable into consistent, perceptually convincing 3D scene geometry. To that end, we adapt direct reward finetuning, a popular technique for human preference alignment. We evaluate the proposed VIST3A approach with different video generators and 3D reconstruction models. All tested pairings markedly improve over prior text-to-3D models that output Gaussian splats. Moreover, by choosing a suitable 3D base model, VIST3A also enables high-quality text-to-pointmap generation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。