用单张照片生成复杂3D发型,首次实现卷发细节还原
DiffLocks: Generating 3D Hair from a Single Image using Diffusion Models
- 基于扩散-变压器架构,直接从图像预测每根头发的三维位置
- 构建4万例合成发型数据集,支持多样式重建
- 无需后处理,适合真实场景照片生成复杂卷发
本文针对从单张图像生成3D发型这一挑战性任务,提出DiffLocks框架。由于发型多样性高且缺乏配对的图像-3D发型数据,现有方法多依赖合成数据并使用低维中间表示(如引导丝和头皮嵌入),需额外后处理才能生成真实感结果,难以恢复细节或处理卷发。为此,我们自动化构建了迄今最大的合成发型数据集,包含40,000个不同发型。利用该数据训练了一个图像条件扩散-变压器模型,可直接从单张正面图像生成精确的3D发丝。通过预训练图像主干网络,模型能泛化至真实场景图像。模型输出一个头皮纹理图,其中每个点包含一根发丝的潜在编码,直接解码为3D结构,无需后处理。这种对单根发丝的显式建模使模型首次成功从单图重建高度卷曲的afro发型。代码与数据已公开。
原文摘要 · Abstract (English)
We address the task of generating 3D hair geometry from a single image, which is challenging due to the diversity of hairstyles and the lack of paired image-to-3D hair data. Previous methods are primarily trained on synthetic data and cope with the limited amount of such data by using low-dimensional intermediate representations, such as guide strands and scalp-level embeddings, that require post-processing to decode, upsample, and add realism. These approaches fail to reconstruct detailed hair, struggle with curly hair, or are limited to handling only a few hairstyles. To overcome these limitations, we propose DiffLocks, a novel framework that enables detailed reconstruction of a wide variety of hairstyles directly from a single image. First, we address the lack of 3D hair data by automating the creation of the largest synthetic hair dataset to date, containing 40K hairstyles. Second, we leverage the synthetic hair dataset to learn an image-conditioned diffusion-transfomer model that generates accurate 3D strands from a single frontal image. By using a pretrained image backbone, our method generalizes to in-the-wild images despite being trained only on synthetic data. Our diffusion model predicts a scalp texture map in which any point in the map contains the latent code for an individual hair strand. These codes are directly decoded to 3D strands without post-processing techniques. Representing individual strands, instead of guide strands, enables the transformer to model the detailed spatial structure of complex hairstyles. With this, DiffLocks can recover highly curled hair, like afro hairstyles, from a single image for the first time. Data and code is available at https://radualexandru.github.io/difflocks/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。