用分部感知知识迁移，高效生成高质量奇幻3D动物。
DreamBeast: Distilling 3D Fantastical Animals with Part-Aware Knowledge Transfer
- 将SD3模型的分部理解能力迁移到3D隐式表示中
- 实现任意视角下即时生成分部亲和图，提升生成质量
- 兼顾生成效果与效率，适合游戏/影视资产创作
我们提出DreamBeast，一种基于得分蒸馏采样（SDS）的新方法，用于生成由不同部件组成的奇幻3D动物资产。现有SDS方法因对文本到图像扩散模型中的部件级语义理解有限，难以胜任该任务。尽管近期扩散模型如Stable Diffusion 3展现出更优的部件级理解能力，但其计算成本过高，且存在单视角扩散模型的普遍问题。DreamBeast通过一种新颖的分部感知知识迁移机制克服这一限制。对于每个生成资产，我们高效地从Stable Diffusion 3模型中提取部件级知识，并构建3D部件亲和隐式表示。该表示支持从任意相机视角快速生成部件亲和图，并在多视角扩散模型的SDS过程中用于引导生成，从而创建具有用户指定部件组合的3D奇幻动物。大量定量与定性评估表明，DreamBeast显著提升了生成3D生物的质量，同时降低了计算开销。
原文摘要 · Abstract (English)
We present DreamBeast, a novel method based on score distillation sampling (SDS) for generating fantastical 3D animal assets composed of distinct parts. Existing SDS methods often struggle with this generation task due to a limited understanding of part-level semantics in text-to-image diffusion models. While recent diffusion models, such as Stable Diffusion 3, demonstrate a better part-level understanding, they are prohibitively slow and exhibit other common problems associated with single-view diffusion models. DreamBeast overcomes this limitation through a novel part-aware knowledge transfer mechanism. For each generated asset, we efficiently extract part-level knowledge from the Stable Diffusion 3 model into a 3D Part-Affinity implicit representation. This enables us to instantly generate Part-Affinity maps from arbitrary camera views, which we then use to modulate the guidance of a multi-view diffusion model during SDS to create 3D assets of fantastical animals. DreamBeast significantly enhances the quality of generated 3D creatures with user-specified part compositions while reducing computational overhead, as demonstrated by extensive quantitative and qualitative evaluations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。