Harmonia优化RAG服务，提升吞吐量并减少延迟违规
Harmonia: End-to-End RAG Serving Optimization
- 通过灵活工作流接口定制RAG处理流程
- 部署多组件分布式系统，吞吐量提升2.04倍以上
- 实时监控自动扩缩容，降低78.4%的SLO违规
检索增强生成（RAG）通过整合外部知识提升大模型可靠性，但服务过程中请求需跨大语言模型推理、数据库与CPU处理等异构组件，效率难保障。我们提出Harmonia，一个端到端RAG服务框架，通过三方面优化：(i) 灵活的工作流定义接口以构建自定义流程；(ii) 考虑异构性的部署机制，将组件作为分布式推理系统配置；(iii) 闭环运行时控制器，根据负载和执行进度动态调整请求优先级与自动扩缩容。在四个RAG应用中，Harmonia性能优于商业方案，吞吐量提升超过2.04倍，SLO违规减少最多达78.4%。
原文摘要 · Abstract (English)
Retrieval-Augmented Generation (RAG) improves the reliability of large language models by integrating external knowledge, but serving RAG pipelines efficiently is challenging because requests traverse heterogeneous components spanning LLM inference, databases, and CPU-side processing. We present Harmonia, an end-to-end RAG serving framework that addresses these bottlenecks through (i) a flexible pipeline specification interface for composing custom workflows, (ii) heterogeneity-aware deployment that provisions and configures components as a distributed inference system, and (iii) a closed-loop runtime controller that monitors load and execution progress and reduces SLO violations through request prioritization and auto-scaling. Across four RAG applications, Harmonia outperforms commercial alternatives, improving throughput by more than 2.04x while reducing SLO violations by up to 78.4 percent.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。