提出FLAME模型,精准预测移动边缘设备的推理延迟。
Taming Asynchronous CPU-GPU Coupling for Frequency-aware Latency Estimation on Mobile Edge
- 分层建模异步耦合,量化并聚合流水线阻塞
- 对DNN和小语言模型仅需分钟级采样,误差小
- 适合需要低延迟与节能的移动边缘应用
精确估计模型推理延迟对时间敏感的移动边缘应用至关重要,可帮助设备在截止时间前预留余量以提升性能或节省资源。然而,动态电压频率调节(DVFS)导致传统静态分析失效,因延迟随CPU/GPU频率变化而波动。尽管可穷举所有频率组合进行采样,但代价高昂——尤其对上下文长度可变的小语言模型(SLMs),耗时可达数日。我们发现简单缩放模型无法准确预测此类波动,根源在于CPU(内核启动)与GPU(执行)之间的复杂异步耦合。本文提出FLAME,通过分层建模重叠并行性,并聚合由异步交互引发的动态流水线空泡,实现跨频率组合的高精度延迟估计。该自底向上方法具有广泛适用性,覆盖从传统DNN到SLMs的多种模型。其精确建模支持稀疏采样,使DNN profiling从小时级缩短至分钟级,SLM从天级降至分钟级,同时保持各频率下小误差。进一步验证表明,基于FLAME的截止时间感知DVFS策略,在能效与延迟保障方面均优于现有最优方法。
原文摘要 · Abstract (English)
Precise estimation of model inference latency is crucial for time-critical mobile edge applications, enabling devices to calculate latency margins against deadlines and trade them for enhanced model performance or resource savings. However, the ubiquity of Dynamic Voltage and Frequency Scaling (DVFS) renders traditional static profiling invalid in real-world deployments, as inference latency fluctuates with varying processor (CPU and GPU) frequencies. While extensive profiling across frequency combinations is theoretically possible, it is prohibitively expensive, particularly for emerging Small Language Models (SLMs), where variable context lengths explode the profiling up to days. We observe that simple analytic scaling fails to predict these fluctuations due to the complex asynchronous coupling between CPU (kernel launching) and GPU (execution). In this paper, we introduce FLAME to accurately estimate inference latency across frequency combinations. It features a novel layer-wise modeling that quantifies the overlapping parallelism and then aggregates dynamic pipeline bubbles caused by asynchronous processor interactions when extending to the full model. This bottom-up approach ensures generalizability across diverse models from DNNs to SLMs, and its precise modeling allows for profiling a sparse subset of samples, cutting DNN profiling from hours to minutes and SLM profiling from days to mere minutes, while maintaining small estimation errors across frequencies. We further showcase FLAME's utility in a deadline-aware DVFS, outperforming the state-of-the-art approach in both power efficiency and latency guarantees.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。