提出新公平性指标MFC，揭示模型跨领域时道德判断的偏差问题。
Fairness Metric Design Exploration in Multi-Domain Moral Sentiment Classification using Transformer-Based Models
- 设计MFC指标衡量道德分类在跨领域中的稳定性
- 发现权威标签跨域性能下降40%且存在显著公平性差异
- 适合关注伦理模型公平性的研究人员和开发者
在跨领域迁移场景下，基于Transformer的道德情感分类面临公平性挑战。本文使用MFTC和MFRC数据集，在多标签设置中评估BERT与DistilBERT模型。整体性能掩盖了分布不均：从Twitter到Reddit迁移导致微平均F1下降14.9%，而反向仅降1.5%。细粒度分析显示，权威标签存在显著不公平——人口均等差异达0.22-0.23，等机会差异高达0.40-0.41。为此，提出道德公平一致性（MFC）指标，实证表明其与人口均等差异呈完美负相关（rho = -1.000, p < 0.001），且独立于传统性能指标。各标签中，忠诚度一致最高（MFC = 0.96），权威最低（MFC = 0.78）。MFC可作为诊断工具，辅助构建更可靠的跨语言道德推理模型。
原文摘要 · Abstract (English)
Ensuring fairness in natural language processing for moral sentiment classification is challenging, particularly under cross-domain shifts where transformer models are increasingly deployed. Using the Moral Foundations Twitter Corpus (MFTC) and Moral Foundations Reddit Corpus (MFRC), this work evaluates BERT and DistilBERT in a multi-label setting with in-domain and cross-domain protocols. Aggregate performance can mask disparities: we observe pronounced asymmetry in transfer, with Twitter->Reddit degrading micro-F1 by 14.9% versus only 1.5% for Reddit->Twitter. Per-label analysis reveals fairness violations hidden by overall scores; notably, the authority label exhibits Demographic Parity Differences of 0.22-0.23 and Equalized Odds Differences of 0.40-0.41. To address this gap, we introduce the Moral Fairness Consistency (MFC) metric, which quantifies the cross-domain stability of moral foundation detection. MFC shows strong empirical validity, achieving a perfect negative correlation with Demographic Parity Difference (rho = -1.000, p < 0.001) while remaining independent of standard performance metrics. Across labels, loyalty demonstrates the highest consistency (MFC = 0.96) and authority the lowest (MFC = 0.78). These findings establish MFC as a complementary, diagnosis-oriented metric for fairness-aware evaluation of moral reasoning models, enabling more reliable deployment across heterogeneous linguistic contexts. .
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。