通过稀疏性与计算强度优化边缘设备的DNN推理调度。
SparOA: Sparse and Operator-aware Hybrid Scheduling for Edge DNN Inference
- 基于阈值预测与强化学习动态调度算子执行。
- 相比基线平均提速1.22-1.31倍，比纯CPU快50.7倍。
- 适合资源受限边缘场景，兼顾性能与能效。
深度神经网络（DNN）的资源需求在资源受限的边缘设备上带来显著性能挑战。现有方案如模型压缩常牺牲精度，专用硬件则成本高且不灵活。混合推理方法通常忽略算子特性对性能的影响。本文提出SparOA，一种基于CPU-GPU的混合推理框架，利用稀疏性和计算强度优化算子调度。SparOA包含三个核心组件：(1) 精准判断最优稀疏度与计算强度阈值的阈值预测器；(2) 基于强化学习、根据实时硬件状态动态优化资源分配的调度器；(3) 通过异步执行和批处理大小优化提升效率的混合推理引擎。大量实验表明，SparOA相比所有基线平均提速1.22-1.31倍，相比纯CPU最高提速50.7倍，并在每推理能耗上达到最优，较当前最优共执行基线节省7%-16%能耗。
原文摘要 · Abstract (English)
The resource demands of deep neural network (DNN) models introduce significant performance challenges, especially when deployed on resource-constrained edge devices. Existing solutions like model compression often sacrifice accuracy, while specialized hardware remains costly and inflexible. Hybrid inference methods, however, typically overlook how operator characteristics impact performance. In this work, we present SparOA, a CPU-GPU hybrid inference framework, which leverages both sparsity and computational intensity to optimize operator scheduling. SparOA embraces aforementioned challenges through three key components: (1) a threshold predictor that accurately determines optimal sparsity and computational intensity thresholds; (2) a reinforcement learning-based scheduler that dynamically optimizes resource allocation based on real-time hardware states; and (3) a hybrid inference engine that enhances efficiency through asynchronous execution and batch size optimization.Extensive results show that SparOA achieves an average speedup of 1.22-1.31x compared to all baselines, and outperforms the CPU-Only by up to 50.7x. Also, SparOA achieves optimal energy-per-inference, consuming 7\%-16\% less energy than the SOTA co-execution baseline.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。