Twill实时调度AI任务，降低延迟54%且不超功耗
Twill: Scheduling Compound AI Systems on Heterogeneous Mobile Edge Platforms
- 基于任务亲和性与动态电压频率调节的实时调度策略
- 在边缘平台实现平均54%延迟降低，满足功耗约束
- 适合需并发运行DNN与大模型的移动边缘场景
复合人工智能（cAI）系统通过串联多个AI模型解决复杂问题，通常包含深度神经网络（DNN）、Transformer和大语言模型（LLM），具有高度计算多样性与动态负载变化。在移动边缘平台部署cAI服务时，如何调度并发的DNN-Transformer推理任务成为挑战，尤其当任务以未知序列动态到达时。现有边缘AI推理方案仅支持多DNN或仅Transformer工作负载，依赖设计阶段性能分析，无法处理cAI所需的DNN与Transformer混合并发推理。本文提出Twill，一个运行时框架，通过任务亲和性感知的集群映射与迁移、优先级感知的任务冻结/解冻及动态电压频率调节（DVFS），在满足功耗预算的前提下最小化推理延迟。我们在Nvidia Jetson Orin NX平台上实现并部署Twill框架，评估结果表明，相较于先进边缘AI推理技术，在主流DNN与LLM上平均降低54%推理延迟，同时严格遵守功耗限制。
原文摘要 · Abstract (English)
Compound AI (cAI) systems chain multiple AI models to solve complex problems. cAI systems are typically composed of deep neural networks (DNNs), transformers, and large language models (LLMs), exhibiting a high degree of computational diversity and dynamic workload variation. Deploying cAI services on mobile edge platforms poses a significant challenge in scheduling concurrent DNN-transformer inference tasks, which arrive dynamically in an unknown sequence. Existing mobile edge AI inference strategies manage multi-DNN or transformer-only workloads, relying on design-time profiling, and cannot handle concurrent inference of DNNs and transformers required by cAI systems. In this work, we address the challenge of scheduling cAI systems on heterogeneous mobile edge platforms. We present Twill, a run-time framework to handle concurrent inference requests of cAI workloads through task affinity-aware cluster mapping and migration, priority-aware task freezing/unfreezing, and DVFS, while minimizing inference latency within power budgets. We implement and deploy our Twill framework on the Nvidia Jetson Orin NX platform. We evaluate Twill against state-of-the-art edge AI inference techniques over contemporary DNNs and LLMs, reducing inference latency by 54% on average, while honoring power budgets.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。