比较深度模型与大模型在抑郁检测中的偏见,发现大模型更公平且表现更好。
Assessing Algorithmic Bias in Language-Based Depression Detection: A Comparison of DNN and LLM Approaches
- 用大模型少样本学习和深度模型加公平损失函数对比
- 大模型对少数族裔表现更好,性别偏见更低但种族偏见仍存
- 特定提示设计可减小性别偏见,但增加样本数无效
本文研究基于语言的自动抑郁检测模型中的算法偏见,重点关注性别和种族/族裔相关的社会人口差异。比较了基于深度神经网络(DNN)嵌入的模型与使用大语言模型(LLM)的少样本学习方法,在Distress Analysis Interview Corpus/Wizard-of-Oz(DAIC-WOZ)临床访谈转录数据上评估性能与公平性。为缓解偏见,对DNN模型采用公平性感知损失函数,对LLM则探索不同提示框架和样本数(shot count)的上下文学习。结果表明,LLM在抑郁分类上优于DNN模型,尤其在少数群体如西班牙裔参与者中表现更佳;LLM的性别偏见低于基于DNN的嵌入,但种族差异依然存在。在公平性技术中,针对最差群体的损失函数能更好平衡性能与公平性,而公平正则化损失虽降低整体损失但效果较差。对于LLM,带有伦理框架的引导提示可缓解1样本设置下的性别偏见,但增加样本数未能进一步减少差距;对种族而言,提示策略或增加$N$均无法有效缓解偏见。
原文摘要 · Abstract (English)
This paper investigates algorithmic bias in language-based models for automated depression detection, focusing on socio-demographic disparities related to gender and race/ethnicity. Models trained using deep neural networks (DNN) based embeddings are compared to few-shot learning approaches with large language models (LLMs), evaluating both performance and fairness on clinical interview transcripts from the Distress Analysis Interview Corpus/Wizard-of-Oz (DAIC-WOZ). To mitigate bias, fairness-aware loss functions are applied to DNN-based models, while in-context learning with varied prompt framing and shot counts is explored for LLMs. Results indicate that LLMs outperform DNN-based models in depression classification, particularly for underrepresented groups such as Hispanic participants. LLMs also exhibit reduced gender bias compared to DNN-based embeddings, though racial disparities persist. Among fairness-aware techniques for mitigating bias in DNN-based embeddings, the worst-group loss, which is designed to minimize loss for the worst-performing demographic group, achieves a better balance between performance and fairness. In contrast, the fairness-regularized loss minimizes loss across all groups but performs less effectively. In LLMs, guided prompting with ethical framing helps mitigate gender bias in the 1-shot setting. However, increasing the number of shots does not lead to further reductions in disparities. For race/ethnicity, neither prompting strategy nor increasing $N$ in $N$-shot learning effectively reduces disparities.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。