arXiv:2506.24045cs.DCcs.LG2025-06被引 9

为手机端智能代理设计了高效调度引擎,兼顾实时响应与后台任务

Agent.xpu: Efficient Scheduling of Agentic LLM Workloads on Heterogeneous SoC

  • 构建异构执行图,动态绑定算子到最优加速器
  • 实现前后台任务并发调度,提升后台吞吐1.2-4.9倍
  • 支持细粒度抢占,确保即时响应且不挤占后台资源

个人大模型代理日益融合前台实时交互与后台主动监控,形成持续运行、带状态的长时序推理流,交替进行预填充和逐令牌解码。尽管现代异构SoC集成CPU、iGPU和NPU以支持本地智能,但现有大模型引擎仍假设静态单次推理,缺乏对流级并发、优先级调度和加速器协同的支持。因此,通用SoC难以适配个人代理动态、混合关键性的执行模式。本文提出Agent.xpu,首个在通用SoC上协调并发前台与后台大模型流的引擎。深入分析揭示了操作符-加速器亲和性、非对称DDR争用及阶段分异批处理行为等独特特性,不同于云端服务假设。Agent.xpu引入三项关键技术:捕捉NPU/iGPU亲和性的异构执行图(HEG)与弹性算子绑定;具有阶段弹性的流感知NPU-iGPU协同,解耦预填充与解码以降低带宽争用并强制优先级;以及基于空闲时间的细粒度抢占与捎带机制,保障前台响应性而不饿死后台任务。在真实个人代理负载下,相较工业级iGPU单引擎与最优张量划分的NPU-iGPU静态推理方案,Agent.xpu实现至少91%的前台延迟降低,后台吞吐提升1.2-4.9倍,同时通过控制iGPU使用最小化能耗与图形干扰。

原文摘要 · Abstract (English)

Personal LLM agents increasingly combine foreground reactive interactions with background proactive monitoring, forming long-lived, stateful LLM flows that interleave prefill and token-by-token decode. While modern heterogeneous SoCs integrate CPUs, iGPUs, and NPUs to support on-device intelligence, existing LLM engines assume static, single-shot inference and lack mechanisms for flow-level concurrency, prioritization, and efficient accelerator coordination. As a result, commodity SoCs remain poorly matched to the dynamic, mixed-criticality execution patterns of personal agents. This paper presents Agent$.$xpu, the first LLM engine that orchestrates concurrent reactive and proactive LLM flows on commodity SoCs. Extensive profiling uncovers unique SoC characteristics of operator-accelerator affinity, asymmetric DDR contention, and stage-divergent batching behaviors distinct from cloud-serving assumptions. Agent$.$xpu introduces three key techniques: a heterogeneous execution graph (HEG) capturing NPU/iGPU affinity and elastic operator binding; flow-aware NPU-iGPU coordination with stage elasticity, decoupling prefill and decode to reduce bandwidth contention and enforce priorities; and fine-grained preemption with slack-aware piggybacking to guarantee reactive responsiveness without starving proactive work. Across realistic personal-agent workloads, Agent$.$xpu delivers 1.2-4.9$\times$ proactive throughput and reduces reactive latency by at least 91%, compared with both industrial iGPU-only serving engine and NPU-iGPU static inference with optimal tensor-partitioning schemes. Agent$.$xpu also minimizes energy consumption and graphics interference via controlled iGPU usage.

大模型调度异构计算边缘智能低延迟

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。