arXiv:2411.12002cs.CV2024-11被引 1

解决3D人脸生成中肤色不一致与偏见问题,提升光照还原真实性。

Analyzing and Improving the Skin Tone Consistency and Bias in Implicit 3D Relightable Face Generators

  • 通过归一化SH系数消除亮度偏差,统计对齐方向分量。
  • 使不同肤色样本生成的面部光影更一致,尤其改善深肤色图像表现。
  • 适合关注生成公平性与真实感的人脸生成研究者。

随着生成对抗网络(GAN)和神经渲染的发展,3D可重光照人脸生成受到广泛关注。现有方法通常采用隐式光照表示,通过合成反照率与依赖光照的阴影图相乘生成重光照图像。尽管该方法能生成细节丰富的阴影效果,但在从深肤色个体图像中提取光照时,常导致肤色不一致,且倾向于生成较浅肤色的反照率图。我们发现该问题源于训练中使用的球谐函数(SH)系数存在偏差,不仅出现在0阶(直流项),其他阶次也存在偏差。为此,我们提出一种简单有效的方法:以直流项归一化SH系数,消除幅度偏差;同时对其他阶次系数进行统计对齐,缓解方向偏差;并引入缩放策略,使生成图像的光照幅度分布与训练数据匹配。大量实验证明,该方法显著提升了肤色一致性,有效缓解了生成偏见。

原文摘要 · Abstract (English)

With the advances in generative adversarial networks (GANs) and neural rendering, 3D relightable face generation has received significant attention. Among the existing methods, a particularly successful technique uses an implicit lighting representation and generates relit images through the product of synthesized albedo and light-dependent shading images. While this approach produces high-quality results with intricate shading details, it often has difficulty producing relit images with consistent skin tones, particularly when the lighting condition is extracted from images of individuals with dark skin. Additionally, this technique is biased towards producing albedo images with lighter skin tones. Our main observation is that this problem is rooted in the biased spherical harmonics (SH) coefficients, used during training. Following this observation, we conduct an analysis and demonstrate that the bias appears not only in band 0 (DC term), but also in the other bands of the estimated SH coefficients. We then propose a simple, but effective, strategy to mitigate the problem. Specifically, we normalize the SH coefficients by their DC term to eliminate the inherent magnitude bias, while statistically align the coefficients in the other bands to alleviate the directional bias. We also propose a scaling strategy to match the distribution of illumination magnitude in the generated images with the training data. Through extensive experiments, we demonstrate the effectiveness of our solution in increasing the skin tone consistency and mitigating bias.

3D生成肤色一致光照建模公平性

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。