提出一种测试时公平性调优方法,让视觉语言模型在不重训练下更公平地应对群体偏差。
Fairness-Aware Test-Time Prompt Tuning

- 通过软提示调优,在测试时同时降低目标熵、提升伪相关熵以增强公平性。
- 在子群体分布偏移下,标准测试时调优会加剧少数群体差异,而新方法避免灾难性遗忘。
- 适合部署在无法重训练且需保护隐私的场景,如医疗或金融AI应用。
视觉语言模型在多模态理解中表现出色,但常对受保护群体产生系统性偏差,且现有方法需重训练并依赖人口属性信息。本文研究了在分布偏移下,基于事件记忆的测试时适应(TTA)对CLIP分类公平性的影响,提出新的公平感知型测试时提示调优方法FairTPT,通过软提示调优联合最小化目标边缘熵、最大化伪相关边缘熵。研究发现:标准测试时调优通常加剧多数与少数群体间的差距;在不损害目标性能的前提下屏蔽伪相关特征本就困难;过度屏蔽会导致模型崩溃。通过监测线性阶段测试时目标损失变化可防止模型崩溃,同时在反应式数据上实现公平性提升并保持整体性能。FairTPT优于所有现有先进测试时去偏方法,为实际应用中的鲁棒测试时适应奠定基础。
原文摘要 · Abstract (English)
Vision-language models have displayed remarkable capabilities in multi-modal understanding and are increasingly used in critical applications where economic and practical deployment constraints prohibit re-training or fine-tuning. However, these models can also exhibit systematic biases that disproportionately affect protected demographic groups and existing approaches to addressing these biases require extensive model retraining and access to demographic attributes. There is a clear need to develop test-time adaptation (TTA) approaches that improve the fairness characteristics of pretrained models under distributional shift. In this paper, we evaluate how episodic TTA affects fairness in CLIP classification under subpopulation shifts and develop FairTPT, a novel fairness-aware episodic TTA method that jointly minimizes target marginal entropy while maximizing spurious marginal entropy through soft-prompt tuning. We find that standard episodic TTA generally exacerbates disparities between majority and minority groups, that blinding a model to spurious attributes without degrading target performance is inherently challenging, and that excessive blinding can lead to catastrophic forgetting. This model collapse can be prevented by monitoring test-time changes in target loss within the linear regime, while still achieving fairness improvements on reactive data and preserving overall performance. FairTPT outperforms all state-of-the-art episodic test-time debiasing methods and establishes a foundation for robust TTA, which is essential for achieving fairness in practice.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。