arXiv:2604.09946cs.CYcs.IR2026-04

通过用户参与式审计,揭示搜索排名系统隐藏的负面影响

All Eyes on the Ranker: Participatory Auditing to Surface Blind Spots in Ranked Search Results

论文配图:All Eyes on the Ranker: Participatory Auditing to Surface Blind Spots in Ranked Search Results
图 1 · 摘自论文原文
  • 设计三场用户工作坊,对比传统与神经排序模型的使用体验
  • 发现用户对智能排序产生信任后,难以察觉恶意排名操纵
  • 提出四类用户感知影响分类,适合关注算法公平性的研究者

以排序列表呈现搜索结果的搜索引擎是公众获取信息的基础技术。现有评估多由领域专家开展,聚焦模型性能、相关性判断或输出分析,而非用户对问责、伤害和信任的真实体验。本文主张参与式审计对揭示用户对排名结果因果与情境理解至关重要,尤其当排序模型在语义理解上日益精准时。我们开展了三场参与式审计工作坊(n=21),参与者通过定制化搜索界面完成四项任务:对比词法排序器(BM25)与神经语义重排序器(MonoT5),探索不同透明度与用户控制程度,并检视人为恶意操纵的排名结果。反思性活动促使参与者构建从系统特性到广泛影响的因果叙事。综合发现,我们提出一个用户感知影响的分类体系,涵盖认知、表征、基础设施及下游社会影响四类。然而,与神经模型互动中暴露了参与式审计的局限:系统表现越专业,用户越信任,越缺乏批判性审视,导致操纵未被察觉。参与者普遍要求可见完整的搜索流程及申诉机制。这些成果表明,参与式审计可揭示传统评估忽略的问责缺口与用户感知影响,同时也揭示其在高可信系统中的适用边界。

原文摘要 · Abstract (English)

Search engines that present users with a ranked list of search results are a fundamental technology for providing public access to information. Evaluations of such systems are typically conducted by domain experts and focus on model-centric metrics, relevance judgments, or output-based analyses, rather than on how accountability, harm, or trust are experienced by users. This paper argues that participatory auditing is essential for revealing users' causal and contextual understandings of how ranked search results produce impacts, particularly as ranking models appear increasingly convincing and sophisticated in their semantic interpretation of user queries. We report on three participatory auditing workshops (n=21) in which participants engaged with a custom search interface across four tasks, comparing a lexical ranker (BM25) and a neural semantic reranker (MonoT5), exploring varying levels of transparency and user controls, and examining an intentionally adversarially manipulated ranking. Reflexive activities prompted participants to articulate causal narratives linking search system properties to broader impacts. Synthesising the findings, we contribute a taxonomy of user-perceived impacts of ranked search results, spanning epistemic, representational, infrastructural, and downstream social impacts. However, interactions with the neural model revealed limits to participatory auditing itself: perceived system competence and accumulated trust reduced critical scrutiny during the workshop, allowing manipulations to go undetected. Participants expressed desire for visibility into the full search pipeline and recourse mechanisms. Together, these findings show how participatory auditing can surface user perceived impacts and accountability gaps that remain unseen when relying on conventional audits, while revealing where participatory auditing may encounter limitations.

搜索排序用户审计算法透明信任机制

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。