首次实现大模型与检索增强系统多目标超参优化，兼顾成本、速度、安全与对齐。
Faster, Cheaper, Better: Multi-Objective Hyperparameter Optimization for LLM and RAG Systems
- 采用贝叶斯优化联合调优大模型与检索系统全链路参数。
- 在两个新基准任务上显著优于基线，获得更优的权衡曲线。
- 提醒实践者：最优配置不跨任务通用，需针对性调优。
尽管检索增强生成（RAG）已成为提升大语言模型（LLM）系统性能的流行方法，但其引入了大量模型选择、参数与超参数决策，包括LLM、嵌入模型和排序器本身，以及各组件的超参数。然而，由于解空间过大、目标评估噪声高、评估成本高昂，联合优化整个系统配置仍鲜有研究，尤其在多目标场景下。本文提出首个针对完整LLM与RAG系统在成本、延迟、安全与对齐四个目标上的多目标超参数优化方法。实验表明，贝叶斯优化显著优于基线方法，在两个新构建的RAG基准任务上获得更优的帕累托前沿。最后，我们为设计多目标RAG系统的实践者提供重要建议，强调最优配置可能无法跨任务和目标泛化。
原文摘要 · Abstract (English)
While Retrieval Augmented Generation (RAG) has emerged as a popular technique for improving Large Language Model (LLM) systems, it introduces a large number of choices, parameters and hyperparameters that must be made or tuned. This includes the LLM, embedding, and ranker models themselves, as well as hyperparameters governing individual RAG components. Yet, collectively optimizing the entire configuration in a RAG or LLM system remains under-explored - especially in multi-objective settings - due to intractably large solution spaces, noisy objective evaluations, and the high cost of evaluations. In this work, we introduce the first approach for multi-objective parameter optimization of cost, latency, safety and alignment over entire LLM and RAG systems. We find that Bayesian optimization methods significantly outperform baseline approaches, obtaining a superior Pareto front on two new RAG benchmark tasks. We conclude our work with important considerations for practitioners who are designing multi-objective RAG systems, highlighting nuances such as how optimal configurations may not generalize across tasks and objectives.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。