无需重训,自动发现潜空间方向实现多属性公平生成。
FairGen: Enhancing Fairness in Text-to-Image Diffusion Models via Self-Discovering Latent Directions
- 自发现潜空间方向,不依赖人工标注参考数据集。
- 在性别、种族及交叉偏见上均超越现有最佳方法。
- 轻量级插件式设计,适配各类扩散模型且无需重新训练。
尽管扩散模型在图像生成任务中表现卓越,但仍会反映训练数据中的固有偏见。随着扩散模型广泛应用于现实场景,这些偏见可能加剧社会认知扭曲,限制少数群体机会。现有去偏方法通常需通过人工构建参考数据集或额外分类器进行模型重训练,存在两大局限:(1) 构建参考数据集成本高昂;(2) 去偏效果受参考数据集或分类器质量制约。为此,我们提出 FairGen,一种无需重训的即插即用方法,通过自发现方式学习属性潜空间方向,彻底摆脱对参考数据集的依赖。FairGen 包含一组属性适配器和一个分布指示器:每个适配器通过噪声组合的自发现过程优化,以学习特定属性的潜空间方向;分布指示器与适配器集合相乘,引导生成过程趋向预定分布。该方法可同时去偏多个属性,保持轻量且易于集成至其他扩散模型。大量实验表明,在性别、种族及其交叉偏见去偏任务中,FairGen 显著优于当前最先进方法。
原文摘要 · Abstract (English)
While Diffusion Models (DM) exhibit remarkable performance across various image generative tasks, they nonetheless reflect the inherent bias presented in the training set. As DMs are now widely used in real-world applications, these biases could perpetuate a distorted worldview and hinder opportunities for minority groups. Existing methods on debiasing DMs usually requires model retraining with a human-crafted reference dataset or additional classifiers, which suffer from two major limitations: (1) collecting reference datasets causes expensive annotation cost; (2) the debiasing performance is heavily constrained by the quality of the reference dataset or the additional classifier. To address the above limitations, we propose FairGen, a plug-and-play method that learns attribute latent directions in a self-discovering manner, thus eliminating the reliance on such reference dataset. Specifically, FairGen consists of two parts: a set of attribute adapters and a distribution indicator. Each adapter in the set aims to learn an attribute latent direction, and is optimized via noise composition through a self-discovering process. Then, the distribution indicator is multiplied by the set of adapters to guide the generation process towards the prescribed distribution. Our method enables debiasing multiple attributes in DMs simultaneously, while remaining lightweight and easily integrable with other DMs, eliminating the need for retraining. Extensive experiments on debiasing gender, racial, and their intersectional biases show that our method outperforms previous SOTA by a large margin.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。