RAG模型在不同群体查询上表现不公,暴露、效用和归因是关键原因。
Who Benefits from RAG? The Role of Exposure, Utility and Attribution Bias
- 分析检索暴露、文档效用和生成归因三因素对公平性的影响
- 发现RAG放大了不同群体查询的准确率差距
- 适合关注AI公平性与检索增强系统的研究者
增强检索能力的大语言模型(RAG)显著提升了回答准确性,但其公平性仍缺乏研究。本文探讨了三个核心因素对查询群体公平性的影响:群体暴露(检索器决定各群体文档被检索到的比例)、群体效用(文档对答案准确率提升的贡献度)和群体归因(生成器依赖某群体文档的程度)。基于TREC 2022公平排序赛道的三个数据集,针对文章生成与标题生成任务,在四个公平类别中进行实验。结果表明,相比纯LLM,RAG系统加剧了不同群体查询间的平均准确率差异;且暴露、效用与归因与准确率或改进幅度存在强正负相关,凸显其在公平性中的关键作用。数据与代码已开源。
原文摘要 · Abstract (English)
Large Language Models (LLMs) enhanced with Retrieval-Augmented Generation (RAG) have achieved substantial improvements in accuracy by grounding their responses in external documents that are relevant to the user's query. However, relatively little work has investigated the impact of RAG in terms of fairness. Particularly, it is not yet known if queries that are associated with certain groups within a fairness category systematically receive higher accuracy, or accuracy improvements in RAG systems compared to LLM-only, a phenomenon we refer to as query group fairness. In this work, we conduct extensive experiments to investigate the impact of three key factors on query group fairness in RAG, namely: Group exposure, i.e., the proportion of documents from each group appearing in the retrieved set, determined by the retriever; Group utility, i.e., the degree to which documents from each group contribute to improving answer accuracy, capturing retriever-generator interactions; and Group attribution, i.e., the extent to which the generator relies on documents from each group when producing responses. We examine group-level average accuracy and accuracy improvements disparities across four fairness categories using three datasets derived from the TREC 2022 Fair Ranking Track for two tasks: article generation and title generation. Our findings show that RAG systems suffer from the query group fairness problem and amplify disparities in terms of average accuracy across queries from different groups, compared to an LLM-only setting. Moreover, group utility, exposure, and attribution can exhibit strong positive or negative correlations with average accuracy or accuracy improvements of queries from that group, highlighting their important role in fair RAG. Our data and code are publicly available from Github.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。