Abacus通过成本优化提升语义算子系统的性能与效率。
Abacus: A Cost-Based Optimizer for Semantic Operator Systems
- 基于成本模型自动选择最优的语义算子实现方式。
- 在生物医学和法律领域平均提升39.4%质量，成本降低10.8倍，速度提升3.4倍。
- 适用于需平衡质量、成本、延迟的AI文档处理系统开发者。
大语言模型（LLMs）推动了面向海量非结构化文档的新一代数据处理应用发展。多个新编程框架允许开发者通过组合语义算子（如基于LLM的映射、过滤、连接等）构建应用，实现信息抽取、摘要生成等任务。尽管这些语义算子系统在基准测试中表现良好，但其优化仍具挑战性。现有优化器难以同时优化质量、成本或延迟，且多数无法在约束条件下进行权衡。本文提出Abacus，一个可扩展的成本驱动型优化器，可在给定优化目标（可能含约束）下搜索语义算子系统的最佳物理实现。Abacus通过少量验证样本、先验性能假设及LLM评判器估算算子性能。我们在生物医学（BioDEX）、法律（CUAD）文档处理及多模态问答（MMQA）场景中评估Abacus，结果表明：相比次优系统，其平均质量提升6.7%-39.4%，成本降低10.8倍，速度提升3.4倍。
原文摘要 · Abstract (English)
LLMs enable an exciting new class of data processing applications over large collections of unstructured documents. Several new programming frameworks have enabled developers to build these applications by composing them out of semantic operators: a declarative set of AI-powered data transformations with natural language specifications. These include LLM-powered maps, filters, joins, etc. used for document processing tasks such as information extraction, summarization, and more. While systems of semantic operators have achieved strong performance on benchmarks, they can be difficult to optimize. An optimizer for this setting must determine how to physically implement each semantic operator in a way that optimizes the system globally. Existing optimizers are limited in the number of optimizations they can apply, and most (if not all) cannot optimize system quality, cost, or latency subject to constraint(s) on the other dimensions. In this paper we present Abacus, an extensible, cost-based optimizer which searches for the best implementation of a semantic operator system given a (possibly constrained) optimization objective. Abacus estimates operator performance by leveraging a minimal set of validation examples, prior beliefs about operator performance, and/or an LLM judge. We evaluate Abacus on document processing workloads in the biomedical and legal domains (BioDEX; CUAD) and multi-modal question answering (MMQA). We demonstrate that, on-average, systems optimized by Abacus achieve 6.7%-39.4% better quality and are 10.8x cheaper and 3.4x faster than the next best system.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。