arXiv:2606.27559cs.IR2026-06

构建首个敏感信息感知的个人搜索测试集,支持隐私保护检索研究。

A Sensitivity-Aware Test Collection for Search Among Personal Information

  • 基于标注敏感性的Enron邮件数据,人工+LLM生成150个查询与超万条相关性评估。
  • 提供敏感信息分类与感知搜索的基线性能,覆盖稀疏与密集索引方案。
  • 适合隐私保护检索、信息检索评估及大模型应用研究者使用。

传统搜索任务通过返回文档子集来满足用户信息需求,但某些包含有用信息的文档也含敏感个人数据。近年来,敏感信息感知搜索(SAS)模型兴起,旨在提供有效结果的同时避免泄露敏感信息。为此,需具备敏感与非敏感信息、查询及相关性标注的测试集。本文基于真实商业邮件的Enron语料库,对部分文档进行敏感性人工标注,并通过众包生成150个查询和11,471条查询-文档相关性评估。同时,采用大语言模型(LLMs)遵循最佳实践,扩展更多相关性判断与敏感性标签。本文报告了相关性、敏感性分类及敏感信息感知搜索的基线表现,并将该测试集公开,可通过ir_datasets获取,同时在Huggingface提供预建的稀疏与稠密索引,便于研究者直接实验。

原文摘要 · Abstract (English)

Traditional search tasks aim to satisfy user information needs by returning a subset of a collection of documents, ranked by the documents' relevance to a user query. However, some collections that contain useful information also contain sensitive personal information. Recently, there has been increasing interest in the development of Sensitivity-Aware Search (SAS) retrieval models to provide users with effective retrieval results without revealing such sensitive information. To develop such systems, test collections containing both sensitive and non-sensitive information, a set of queries, and query-document relevance assessments are required. The Enron email corpus contains real business-related emails, where some emails also contain sensitive personal information. However, the original Enron collection does not contain queries or query-relevance assessments. To this end, we crowdsource 150 query formulations for 50 different topics and 11,471 query-relevance assessments for a subset of the Enron documents that have been manually labelled for sensitivity. We follow best practices for using large language models (LLMs) in Information Retrieval evaluation to extend the collection further with additional LLM judged query-relevance assessments and sensitivity labels. We present baseline performances for relevance, sensitivity classification, and sensitivity-aware search on the collection. We make the collection available, including through the popular ir_datasets package, and provide pre-built sparse and dense indices on Huggingface to facilitate easy experimentation.

信息检索隐私保护测试集LLM评估

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。