系统评估推荐公平性度量的可靠性,提出改进方法与使用指南。
Offline Evaluation Measures of Fairness in Recommender Systems

- 分析现有公平性度量的理论与实证缺陷,揭示其不可靠根源。
- 提出新评估方法,解决原度量在计算或解释上的根本问题。
- 给出度量选用建议,帮助实际场景中更精准选择工具。
推荐系统公平性评估日益重要,尤其在强调公平人工智能的法规背景下。现有多种公平性度量被提出并使用,但多数缺乏深入分析,导致对其局限性认识不足。例如,尚不清楚何种模型输出会产生最(不)公平评分,度量分数的分布情况如何,以及是否存在无法计算的情况(如除零错误)。这些问题影响了度量结果的解读,也使用户难以选择合适度量。本文通过一系列研究,系统评估并克服现有推荐系统公平性度量在理论、实证和概念层面的局限性。我们针对不同评估主体(用户与物品)和粒度(群体与个体),分析了广泛的离线公平性度量。首先,从理论与实证角度揭示度量在可解释性、表达力与适用性方面的缺陷;其次,提出新的评估方法与度量以解决上述问题;最后,基于度量局限性,提出合理使用指南,助力实践中更精确地选择公平性评估工具。总体而言,本研究推动了推荐系统离线公平性评估的前沿发展。
原文摘要 · Abstract (English)
The evaluation of recommender system fairness has become increasingly important, especially with recent legislation that emphasises the development of fair and responsible artificial intelligence. This has led to the emergence of various fairness evaluation measures, which quantify fairness based on different definitions. However, many of such measures are simply proposed and used without further analysis on their robustness. As a result, there is insufficient understanding and awareness of the measures' limitations. Among other issues, it is not known what kind of model outputs produce the (un)fairest score, how the measure scores are empirically distributed, and whether there are cases where the measures cannot be computed (e.g., due to division by zero). These issues cause difficulty in interpreting the measure scores and confusion on which measure(s) should be used for a specific case. This thesis presents a series of papers that assess and overcome various theoretical, empirical, and conceptual limitations of existing recommender system fairness evaluation measures. We investigate a wide range of offline evaluation measures for different fairness notions, divided based on the evaluation subjects (users and items) and for different evaluation granularities (groups of subjects and individual subjects). Firstly, we perform theoretical and empirical analysis on the measures, exposing flaws that limit their interpretability, expressiveness, or applicability. Secondly, we contribute novel evaluation approaches and measures that overcome these limitations. Finally, considering the measures' limitations, we recommend guidelines for the appropriate measure usage, thereby allowing for more precise selection of fairness evaluation measures in practical scenarios. Overall, this thesis contributes to advancing the state-of-the-art offline evaluation of fairness in recommender systems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。