一句话让任意模态生成3D模型，避免转换损失。
Any-to-3D Generation via Hybrid Diffusion Supervision
- 用跨模态对齐技术统一文本、图像、音频输入
- 新损失函数提升多模态对齐，生成质量显著改善
- 首个支持任意模态输入的3D生成框架，适合多模态研究者
近期3D物体生成进展得益于扩散模型提供的强先验。然而，现有模型仅针对特定任务，一次只能处理一种模态，更换模态需重新训练。若将文本转为图像再输入图像到3D模型，过程耗时且易丢失信息。为此，我们提出XBind，一种基于跨模态预对齐的统一任意模态到3D生成框架。XBind结合多模态对齐编码器与预训练扩散模型，可从文本、图像、音频等任意模态生成3D物体。我们设计了新的模态相似性（MS）损失，对齐模态提示与渲染图像的嵌入，增强3D物体与多模态的一致性。此外，混合扩散监督与三阶段优化流程进一步提升生成质量。大量实验验证了XBind在任意模态到3D场景中的广泛生成能力。据我们所知，这是首个支持任意模态提示生成3D物体的方法。
原文摘要 · Abstract (English)
Recent progress in 3D object generation has been fueled by the strong priors offered by diffusion models. However, existing models are tailored to specific tasks, accommodating only one modality at a time and necessitating retraining to change modalities. Given an image-to-3D model and a text prompt, a naive approach is to convert text prompts to images and then use the image-to-3D model for generation. This approach is both time-consuming and labor-intensive, resulting in unavoidable information loss during modality conversion. To address this, we introduce XBind, a unified framework for any-to-3D generation using cross-modal pre-alignment techniques. XBind integrates an multimodal-aligned encoder with pre-trained diffusion models to generate 3D objects from any modalities, including text, images, and audio. We subsequently present a novel loss function, termed Modality Similarity (MS) Loss, which aligns the embeddings of the modality prompts and the rendered images, facilitating improved alignment of the 3D objects with multiple modalities. Additionally, Hybrid Diffusion Supervision combined with a Three-Phase Optimization process improves the quality of the generated 3D objects. Extensive experiments showcase XBind's broad generation capabilities in any-to-3D scenarios. To our knowledge, this is the first method to generate 3D objects from any modality prompts. Project page: https://zeroooooooow1440.github.io/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。