无需训练即可用单图生成可控3D物体，提升一致性与控制力。
Controllable 3D Object Generation with Single Image Prompt
- 采用现成图像适配器直接生成3D对象，避免额外训练。
- 引入深度条件预热策略，显著提升3D生成一致性。
- 用户测试证实生成结果更贴合输入图像且结构更稳定。
近期扩散模型在图像生成方面展现出惊人的保真度。在快速发展的3D物体生成任务中，现有方法主要依赖文本到图像扩散模型结合文本反转技术，通过训练伪文本提示来描述给定图像。实践中，多种生成模型利用文本反转在伪文本嵌入空间中学习目标物体的概念或风格，从而生成高质量输出。然而，文本反转需额外训练时间，且缺乏控制能力。为此，本文提出两种创新方法：(1) 使用现成图像适配器生成3D物体，无需文本反转，可实现对深度、姿态、文本等条件的增强控制；(2) 提出深度条件预热策略以提升3D一致性。实验表明，本方法在定性和定量评估上与基于文本反转的方法相当，并在3D一致性上表现更优。此外，用户研究显示，本模型在匹配输入图像和保持3D一致性方面优于现有方法，验证了所提方案的有效性。代码已开源至GitHub：https://github.com/Seooooooogi/Control3D_IP/
原文摘要 · Abstract (English)
Recently, the impressive generative capabilities of diffusion models have been demonstrated, producing images with remarkable fidelity. Particularly, existing methods for the 3D object generation tasks, which is one of the fastest-growing segments in computer vision, pre-dominantly use text-to-image diffusion models with textual inversion which train a pseudo text prompt to describe the given image. In practice, various text-to-image generative models employ textual inversion to learn concepts or styles of target object in the pseudo text prompt embedding space, thereby generating sophisticated outputs. However, textual inversion requires additional training time and lacks control ability. To tackle this issues, we propose two innovative methods: (1) using an off-the-shelf image adapter that generates 3D objects without textual inversion, offering enhanced control over conditions such as depth, pose, and text. (2) a depth conditioned warmup strategy to enhance 3D consistency. In experimental results, ours show qualitatively and quantitatively comparable performance and improved 3D consistency to the existing text-inversion-based alternatives. Furthermore, we conduct a user study to assess (i) how well results match the input image and (ii) whether 3D consistency is maintained. User study results show that our model outperforms the alternatives, validating the effectiveness of our approaches. Our code is available at GitHub repository:https://github.com/Seooooooogi/Control3D_IP/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。