arXiv:2605.31575cs.IRcs.AI2026-05

用可控生成框架创建可复现的合成检索数据集,提前暴露系统瓶颈。

SPECTRA: Synthetic IR Test Collections with Relevance Oracles and Controlled Distractor Diagnostics

  • 分离主题结构与文本表达,实现可调控的合成文档生成
  • 生成6万篇文档、961万词,支持96个查询的分级相关性标注
  • 适合评估检索系统性能和失败模式,尤其适用于早期测试阶段

大规模信息检索测试需要足够大的语料库来压测索引构建、排序延迟、查询路由和评估工具链,但人工标注的测试集成本高昂,且在文档私密或仍在设计阶段时难以获取。本文提出SPECTRA,一个可复现的合成文本语料库与检索测试集生成框架,通过分离潜在主题结构、表面文本实现、元数据控制、查询意图生成和确定性相关性判定。该框架旨在作为Cranfield式和TREC式评估的诊断补充,而非替代人工评估。单进程Python原型生成了最多60,000篇文档、961万词的语料,保留可控的长尾词汇增长,并为96个查询生成分级相关性标签。局部仿真研究显示,生成速度保持近线性,约每秒12,000至14,000文档;估计的Zipf斜率绝对值接近0.86;跨主题干扰文本比例从2%增至36%时,BM25 nDCG@10从1.00降至0.43。结果表明,轻量级合成语料可在昂贵的语料构建前有效暴露检索系统的扩展性和失效模式。

原文摘要 · Abstract (English)

Scalable information retrieval testing needs corpora that are large enough to stress index construction, ranking latency, query routing, and evaluation tooling, yet human-judged test collections remain expensive and may be unavailable when documents are private or still under design. This paper introduces SPECTRA, a reproducible framework for generating synthetic text corpora and retrieval test collections through a separation of latent topical structure, surface text realization, metadata controls, query intent generation, and deterministic relevance oracles. The framework is intended as a diagnostic complement to Cranfield-style and TREC-style evaluation, not as a replacement for human assessment. A single-process Python prototype generated corpora up to 60,000 documents and 9.61 million tokens while preserving controllable long-tail vocabulary growth and producing graded relevance labels for 96 queries. In the local simulation study, generation remained close to linear at roughly 12K to 14K documents per second, estimated Zipf slopes stayed near 0.86 in absolute value, and increasing cross-topic distractor text reduced BM25 nDCG@10 from 1.00 at 2% distractors to 0.43 at 36% distractors. These results show that lightweight synthetic corpora can expose retrieval-system scaling and failure modes before costly collection construction begins.

信息检索合成数据评估框架

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。