用可执行程序自动优化文档表示，提升检索效果。
AutoIndex: Learning Representation Programs for Retrieval
- 通过搜索可执行程序自动调整文档表示，而非固定预处理
- 在8个任务上平均提升Recall@100 8.4%、nDCG@10 8.3%
- 适合需要优化检索前处理的系统开发者
我们提出AutoIndex，一种学习表示程序的框架：可执行的变换将原始文档映射为检索系统所用的表示。不同于调优检索器、重排序器或少量预处理超参数，AutoIndex在文档索引前搜索能切片、丰富、归一化、重加权或重组文档的程序。每轮迭代中，AutoIndex进行验证引导的程序搜索，代理诊断当前程序的失败并合成候选更新，仅保留能提升检索质量的更新。我们在CRUMB（一个异构检索任务基准）上评估AutoIndex，所有实验中保持BM25不变。所学程序在全部8项任务上均优于静态全文档BM25基线，平均提升Recall@100达+8.4%，nDCG@10达+8.3%，最大提升分别为+30.5%和+43.6%。结果表明，文档表示不应视为检索前固定的预处理选择，而应作为显式优化目标。代码已开源：https://github.com/auto-index/autoindex。
原文摘要 · Abstract (English)
We present AutoIndex, a framework for learning representation programs: executable transformations that map raw documents into the representations exposed to a retrieval system. Rather than tuning retrievers, rerankers, or a small set of preprocessing hyperparameters, AutoIndex searches over programs that slice, enrich, normalize, reweight, or reorganize documents before indexing. At each iteration, AutoIndex performs validation-guided program search, in which agents diagnose failures of the current program and synthesize candidate updates, retaining only updates that improve retrieval quality under the resulting index. We evaluate AutoIndex on CRUMB, a benchmark of heterogeneous retrieval tasks, with BM25 held fixed across all experiments. The learned programs improve recall over a static full-document BM25 baseline on all 8 tasks, with average gains of +8.4% in Recall@100 and +8.3% in nDCG@10, and largest gains of +30.5% in Recall@100 and +43.6% in nDCG@10. These results suggest that document representation should not be treated as a fixed preprocessing choice made before retrieval begins, but as an explicit optimization target. Code to reproduce our results is available at https://github.com/auto-index/autoindex.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。