揭示智能体工作负载特性,为高效推理系统设计提供实证依据。
From LLM Inference to Agentic Workloads: Characterization and Implications for Serving Systems

- 构建十类智能体应用基准,统一采集系统级运行数据。
- 发现五类应用中非LLM组件主导延迟,会话内存峰值达28GB。
- 提出任务感知、缓存优化等方案,降低延迟40%以上。
智能体应用正将AI服务从单次模型推理转向长周期、需协调工具与持久状态的复杂工作负载。然而,此类工作负载在延迟、成本和瓶颈分布上的系统行为仍缺乏清晰刻画,导致现有服务系统仍依赖传统推理假设。本文提出AgentSysBench,一个包含十种代表性智能体应用的基准套件与测量工具,支持受控部署与生产日志分析。研究发现六类关键差异:(1)执行重且有状态,5/10应用中非LLM组件主导延迟,沙盒工作集内存峰值达28 GB/会话;(2)组件资源需求异构——GPU密集型推理、内存密集型检索、CPU密集型沙盒,任务延迟差距达32倍;(3)瓶颈随请求、模型与部署动态变化;(4)生产会话中状态空闲时长达数分钟至数小时;(5)控制平面开销(如工具调用、上下文冗余)挤占有效计算资源;(6)三个应用的生产数据揭示搜索与网络请求存在大量跨请求冗余,具显著缓存潜力。四类设计探索验证:任务感知调度降低延迟29–40%,通信感知部署提升4.5倍性能,状态外移减少4.6倍内存占用,工具结果缓存消除35.2%冗余搜索调用,节省19.3%总搜索延迟。
原文摘要 · Abstract (English)
Agentic applications are shifting AI serving from isolated model inference to long-running workloads in which LLMs coordinate tools, environments, and persistent state. However, the system behavior of these workloads---where latency, cost, and bottlenecks arise---remains poorly characterized, leaving serving systems to rely on assumptions built for conventional inference. We present AgentSysBench, a benchmark suite and measurement toolkit with ten representative agentic applications and unified systems-level instrumentation. Across controlled deployments and production traces, we identify six properties that distinguish agentic workloads from conventional LLM serving: (1) execution is heavyweight and stateful, with non-LLM components dominating latency in 5 of 10 applications and sandbox working-set memory peaking at 28 GB per session; (2) applications compose components with heterogeneous resource affinity---GPU-bound inference, memory-bound retrieval, CPU-bound sandboxes---whose task latencies diverge by up to 32x; (3) bottlenecks shift across requests, models, and deployments; (4) production sessions hold state idle for minutes to hours between active steps; (5) a control-plane tax---auxiliary LLM calls and context overhead from tool schemas and observations---crowds out productive compute and context; and (6) production traces from three applications reveal heavy cross-request redundancy in search queries and web fetches, exposing a large caching opportunity. Four design explorations demonstrate that these findings are actionable: task-aware serving reduces latency by 29--40%, communication-aware placement by up to 4.5x, state offloading reduces memory usage by 4.6x, and tool-result caching removes 35.2% of redundant search calls and saves 19.3% of aggregate search latency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。