用声明式方法简化纵向检索实验的代码复杂度
Simplified Longitudinal Retrieval Experiments: A Case Study on Query Expansion and Document Boosting
- 通过扩展ir_datasets实现纵向检索的声明式描述
- 在LongEval 2024实验中代码复杂度显著降低
- 适合关注可复现性与可扩展性的检索研究者
纵向检索评估旨在捕捉信息需求与文档随时间的演变。然而，传统的Cranfield式检索评估仅包含静态的查询与文档集，因而忽略了时间维度。因此，纵向评估需在检索工具包中加入自定义逻辑，这增加了研究软件的复杂性，可能降低实验的可复现性和可扩展性。基于我们对2024年LongEval的提交，我们提出了ir_datasets的自定义扩展，支持以声明式方式而非命令式方式描述纵向检索实验的关键方面，例如在何时可用哪些查询、文档或相关性反馈。我们使用新扩展重实现了对LongEval 2024的提交，发现声明式访问可有效降低代码复杂度。
原文摘要 · Abstract (English)
The longitudinal evaluation of retrieval systems aims to capture how information needs and documents evolve over time. However, classical Cranfield-style retrieval evaluations only consist of a static set of queries and documents and thereby miss time as an evaluation dimension. Therefore, longitudinal evaluations need to complement retrieval toolkits with custom logic. This custom logic increases the complexity of research software, which might reduce the reproducibility and extensibility of experiments. Based on our submissions to the 2024 edition of LongEval, we propose a custom extension of ir_datasets for longitudinal retrieval experiments. This extension allows for declaratively, instead of imperatively, describing important aspects of longitudinal retrieval experiments, e.g., which queries, documents, and/or relevance feedback are available at which point in time. We reimplement our submissions to LongEval 2024 against our new ir_datasets extension, and find that the declarative access can reduce the complexity of the code.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。