对比五种超参优化方法，发现贪心和随机搜索可高效提升RAG性能。
An Analysis of Hyper-Parameter Optimization Methods for Retrieval Augmented Generation
- 采用五种超参优化算法，在五个跨领域数据集上测试。
- 贪心优化先选模型比按流程顺序优化效果更好，性能显著提升。
- 无需全量网格搜索，随机或贪心策略即可高效找到最优配置。
为特定任务优化检索增强生成（RAG）配置是一项复杂且资源密集的挑战。尽管近期出现了若干RAG超参数优化（HPO）框架，但其有效性尚未得到严格评估。为此，我们开展了一项全面研究，涵盖五种HPO算法在五个来自不同领域的数据集上的实验，包括一个新整理的真实产品文档数据集。本研究探索了迄今最大的RAG HPO搜索空间，包含完整的网格搜索评估，并以三个评价指标作为优化目标。结果分析表明，RAG HPO可高效完成，无论采用贪心策略还是随机搜索，均能显著提升所有数据集上的RAG性能。对于贪心方法，我们发现优先优化模型选择优于按RAG流程顺序进行优化的常见做法。
原文摘要 · Abstract (English)
Optimizing Retrieval-Augmented Generation (RAG) configurations for specific tasks is a complex and resource-intensive challenge. Motivated by this challenge, frameworks for RAG hyper-parameter optimization (HPO) have recently emerged, yet their effectiveness has not been rigorously benchmarked. To fill this gap, we present a comprehensive study involving five HPO algorithms over five datasets from diverse domains, including a newly curated real-world product documentation dataset. Our study explores the largest RAG HPO search space to date that includes full grid-search evaluations, and uses three evaluation metrics as optimization targets. Analysis of the results shows that RAG HPO can be done efficiently, either greedily or with random search, and that it significantly boosts RAG performance for all datasets. For greedy HPO approaches, we show that optimizing model selection first is preferable to the common practice of following the RAG pipeline order during optimization.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。