用多视角图像重建动物毛发,首次实现跨物种高保真建模。
NeuralFur: Animal Fur Reconstruction From Multi-View Images
- 基于视觉语言模型获取毛发结构先验,指导毛发生长方向。
- 结合几何与光照损失,在多视角图像下实现逼真毛发重建。
- 无需特定数据集,适用于不同种类动物的毛发建模。
从多视角图像中重建真实动物毛发几何结构极具挑战,原因在于毛发的精细细节、自遮挡及视角依赖性外观。与人类发型重建不同,目前尚无可用于学习不同动物毛发先验的数据集。本文提出首个基于多视角图像的动物毛发高保真3D建模方法,采用丝状表示,利用视觉语言模型(VLM)的通用知识。给定多视角RGB图像,首先使用传统多视图立体技术重建粗略表面几何。随后,通过VLM检索身体各部位毛发的真实长度与结构信息,构建无毛几何体,并在其上生长毛发丝。毛发重建同时受几何与光度损失监督,基于多视角图像计算。为缓解输入图像经Gabor滤波导致的方向歧义,额外利用VLM引导毛发生长方向,并引入重力向量作为约束损失。该新范式使模型在多种不同毛发类型的动物间实现良好泛化。更多结果与代码见 https://neuralfur.is.tue.mpg.de。
原文摘要 · Abstract (English)
Reconstructing realistic animal fur geometry from images is a challenging task due to the fine-scale details, self-occlusion, and view-dependent appearance of fur. In contrast to human hairstyle reconstruction, there are also no datasets that can be leveraged to learn a fur prior for different animals. In this work, we present a first multi-view-based method for high-fidelity 3D fur modeling of animals using a strand-based representation, leveraging the general knowledge of a vision language model. Given multi-view RGB images, we first reconstruct a coarse surface geometry using traditional multi-view stereo techniques. We then use a vision language model (VLM) system to retrieve information about the realistic length structure of the fur for each part of the body. We use this knowledge to construct the animal's furless geometry and grow strands atop it. The fur reconstruction is supervised with both geometric and photometric losses computed from multi-view images. To mitigate orientation ambiguities stemming from the Gabor filters that are applied to the input images, we additionally utilize the VLM to guide the strands' growth direction and their relation to the gravity vector that we incorporate as a loss. With this new schema of using a VLM to guide 3D reconstruction from multi-view inputs, we show generalization across a variety of animals with different fur types. For additional results and code, please refer to https://neuralfur.is.tue.mpg.de.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。