首个大规模语言程序评测基准，评估模型组合与优化策略的性价比
LangProBe: a Language Programs Benchmark
- 构建2000+组合的多步语言程序评测框架
- 优化后程序在成本与质量间实现帕累托改进
- 适合研究AI系统架构与自动优化的开发者
将语言模型（LM）组合成多步骤语言程序并自动优化模块化提示，已成为构建AI系统的主流范式，但该领域中的权衡关系此前研究极少。我们提出LangProBe，首个大规模语言程序评测基准，涵盖2000多个任务、架构、优化器和语言模型的组合。利用该基准，我们首次系统研究了程序架构与优化器（及其组合及与不同模型的交互）对质量与成本权衡的影响。结果表明，优化后的语言程序相比直接调用模型，在成本与质量间实现显著帕累托改进，但最佳组合仍需依赖人工判断或经验决策。我们将开源LangProBe的代码与评测数据。
原文摘要 · Abstract (English)
Composing language models (LMs) into multi-step language programs and automatically optimizing their modular prompts is now a mainstream paradigm for building AI systems, but the tradeoffs in this space have only scarcely been studied before. We introduce LangProBe, the first large-scale benchmark for evaluating the architectures and optimization strategies for language programs, with over 2000 combinations of tasks, architectures, optimizers, and choices of LMs. Using LangProBe, we are the first to study the impact of program architectures and optimizers (and their compositions together and with different models) on tradeoffs of quality and cost. We find that optimized language programs offer strong cost--quality Pareto improvement over raw calls to models, but simultaneously demonstrate that human judgment (or empirical decisions) about which compositions to pursue is still necessary for best performance. We will open source the code and evaluation data for LangProBe.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。