构建可扩展推理架构，显著降低智能体系统延迟与成本。
Scalable Inference Architectures for Compound AI Systems: A Production Deployment Study
- 采用无服务器+动态扩缩容+MLOps，支持多模型并行调用。
- 尾延迟降低50%以上，吞吐提升3.9倍，成本节省30%-40%。
- 适合企业级智能体系统部署，支持快速迭代与突发负载。
现代企业级AI应用越来越多依赖复合型AI系统——由多个模型、检索器和工具组合完成复杂任务。在生产环境中部署此类系统，需具备高效服务并发、异构模型调用的能力，同时保证低延迟与成本效益。本文报告了Salesforce开发的模块化、平台无关的推理架构在生产环境中的实践，支持Agentforce（自主AI智能体）和ApexGuru（AI代码分析）等复合型AI场景。系统融合无服务器执行、动态自动扩缩容与MLOps流水线，实现多组件智能体工作流中一致的低延迟推理。生产数据显示，尾延迟（P95）降低超50%，吞吐最高提升3.9倍，成本节约30%至40%，优于以往静态部署。进一步分析揭示了复合系统特有的挑战：多模型分发开销、级联冷启动传播、异构扩缩动态等，在服务智能体工作负载时尤为突出。通过详细案例与运维经验，说明该架构如何实现模型调用并行化、应对突发多智能体负载，并支持快速模型迭代，为企业在大规模部署智能体AI提供关键支撑。
原文摘要 · Abstract (English)
Modern enterprise AI applications increasingly rely on compound AI systems - architectures that compose multiple models, retrievers, and tools to accomplish complex tasks. Deploying such systems in production demands inference infrastructure that can efficiently serve concurrent, heterogeneous model invocations while maintaining cost-effectiveness and low latency. This paper presents a production deployment study of a modular, platform-agnostic inference architecture developed at Salesforce to support compound AI use cases including Agentforce (autonomous AI agents) and ApexGuru (AI-powered code analysis). The system integrates serverless execution, dynamic autoscaling, and MLOps pipelines to deliver consistent low-latency inference across multi-component agent workflows. We report production results demonstrating over 50% reduction in tail latency (P95), up to 3.9x throughput improvement, and 30 to 40% cost savings compared to prior static deployments. We further present a novel analysis of compound-system-specific challenges including multi-model fan-out overhead, cascading cold-start propagation, and heterogeneous scaling dynamics that emerge uniquely when serving agentic workloads. Through detailed case studies and operational lessons, we illustrate how the architecture enables compound AI systems to scale model invocations in parallel, handle bursty multi-agent workloads, and support rapid model iteration - capabilities essential for operationalizing agentic AI at enterprise scale.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。