对比14种检索器，发现大模型检索器效率与收益不匹配。
Are LLM-Based Retrievers Worth Their Cost? An Empirical Study of Efficiency, Robustness, and Reasoning Overhead
- 在12个任务上测试14种检索器，评估延迟、吞吐和鲁棒性
- 推理增强使小模型延迟几乎不变，但顶尖模型增益递减
- 检索得分可信度低，需额外校准才能用于下游路由
大型语言模型检索器虽提升复杂查询表现，但其实际价值取决于效率、鲁棒性及可靠置信度信号。我们在12个任务和14种检索器上复现了高推理强度的检索基准（BRIGHT），并扩展评估冷启动索引成本、查询延迟分布与吞吐量、语料库缩放、可控查询扰动下的鲁棒性，以及预测查询成功率的置信度（AUROC）。通过比较标准查询与五种推理增强变体，量化了推理开销：准确率提升与附加延迟的权衡。结果表明，部分专为推理设计的检索器在保持高吞吐的同时表现优异；而多个基于大模型的双编码器检索器延迟显著，收益却有限。推理增强对<10亿参数编码器影响极小，但对顶级检索器回报递减，且在正式数学/代码领域可能降低性能。各类模型的置信度校准普遍不佳，原始检索得分不可靠，无法直接用于下游路由，须额外校准。所有代码与数据已开源。
原文摘要 · Abstract (English)
Large language model retrievers improve performance on complex queries, but their practical value depends on efficiency, robustness, and reliable confidence signals in addition to accuracy. We reproduce a reasoning-intensive retrieval benchmark (BRIGHT) across 12 tasks and 14 retrievers, and extend evaluation with cold-start indexing cost, query latency distributions and throughput, corpus scaling, robustness to controlled query perturbations, and confidence use (AUROC) for predicting query success. We also quantify \emph{reasoning overhead} by comparing standard queries to five provided reasoning-augmented variants, measuring accuracy gains relative to added latency. We find that some reasoning-specialized retrievers achieve strong effectiveness while remaining competitive in throughput, whereas several large LLM-based bi-encoders incur substantial latency for modest gains. Reasoning augmentation incurs minimal latency for sub-1B encoders but exhibits diminishing returns for top retrievers and may reduce performance on formal math/code domains. Confidence calibration is consistently weak across model families, indicating that raw retrieval scores are unreliable for downstream routing without additional calibration. We release all code and artifacts for reproducibility.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。