提出基于感知的肤色评估方法,提升面部情绪分析公平性评估可靠性
TrustSkin: A Fairness Pipeline for Trustworthy Facial Affect Analysis Across Skin Tone
- 采用亮度与色相(L*-H*)替代传统ITA,减少光照干扰影响
- 暗肤色群体占比仅2%,模型在该组F1-score差距达0.08,真阳性率差0.11
- 提供可插拔公平性分析流程,适合关注算法公平性的研究者使用
评估面部情绪分析(FAA)系统在不同人群中的表现,需可靠测量敏感属性如族源,常以肤色近似表示,但肤色受光照显著影响。本研究比较了两种客观肤色分类方法:广泛应用的个体分型角(ITA)与基于感知的亮度(L*)和色相(H*)方法。基于AffectNet数据集和MobileNet模型,评估两种方法定义的肤色组别间的公平性。结果发现,暗肤色群体仅占约2%,且不同组别间存在严重不公平现象:F1-score差异最高达0.08,真阳性率(TPR)差异最高达0.11。尽管ITA受光照影响大,而L*-H*方法能更稳定地划分子组,并通过等机会等指标实现更清晰诊断。Grad-CAM分析显示模型对不同肤色的关注模式存在差异,暗示特征编码不一致。为此,我们提出一个模块化公平性增强管道,集成感知肤色估计、模型可解释性与公平性评估。研究强调肤色测量方式对公平性评估的重要性,指出依赖ITA的评估可能掩盖对深肤色群体的影响。
原文摘要 · Abstract (English)
Understanding how facial affect analysis (FAA) systems perform across different demographic groups requires reliable measurement of sensitive attributes such as ancestry, often approximated by skin tone, which itself is highly influenced by lighting conditions. This study compares two objective skin tone classification methods: the widely used Individual Typology Angle (ITA) and a perceptually grounded alternative based on Lightness ($L^*$) and Hue ($H^*$). Using AffectNet and a MobileNet-based model, we assess fairness across skin tone groups defined by each method. Results reveal a severe underrepresentation of dark skin tones ($\sim 2 \%$), alongside fairness disparities in F1-score (up to 0.08) and TPR (up to 0.11) across groups. While ITA shows limitations due to its sensitivity to lighting, the $H^*$-$L^*$ method yields more consistent subgrouping and enables clearer diagnostics through metrics such as Equal Opportunity. Grad-CAM analysis further highlights differences in model attention patterns by skin tone, suggesting variation in feature encoding. To support future mitigation efforts, we also propose a modular fairness-aware pipeline that integrates perceptual skin tone estimation, model interpretability, and fairness evaluation. These findings emphasize the relevance of skin tone measurement choices in fairness assessment and suggest that ITA-based evaluations may overlook disparities affecting darker-skinned individuals.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。