提出以最差情况性能评估信息获取系统，更符合公平与社会价值。
Pessimistic Evaluation
- 关注系统在最不利情况下的表现，而非平均性能
- 在检索与推荐任务中验证了方法的有效性
- 适合关注公平性与社会影响的研究者
传统信息获取系统的评估主要基于多个信息需求（信息检索）或用户群体（推荐系统）的平均效用。本文认为，仅依赖平均指标的评估方式隐含功利主义价值观，与信息平等获取的传统理念不符。为此，我们倡导采用悲观评估方法，聚焦最差情况下的系统效用。该方法（a）植根于伦理与实用概念，（b）理论上可与现有鲁棒性与公平性方法互补，（c）在多种检索与推荐任务中得到实证验证。结果表明，将悲观评估纳入现有实验流程，有助于更全面理解系统行为，尤其在关注社会福祉时具有重要意义。
原文摘要 · Abstract (English)
Traditional evaluation of information access systems has focused primarily on average utility across a set of information needs (information retrieval) or users (recommender systems). In this work, we argue that evaluating only with average metric measurements assumes utilitarian values not aligned with traditions of information access based on equal access. We advocate for pessimistic evaluation of information access systems focusing on worst case utility. These methods are (a) grounded in ethical and pragmatic concepts, (b) theoretically complementary to existing robustness and fairness methods, and (c) empirically validated across a set of retrieval and recommendation tasks. These results suggest that pessimistic evaluation should be included in existing experimentation processes to better understand the behavior of systems, especially when concerned with principles of social good.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。