通过层变体调度，提升异构加速器上多模型实时推理的准时率。
Terastal: Layer-Variant-based Scheduling for Real-Time Multi-DNN Workloads on Heterogeneous Accelerators
- 为每层设计适配非优选加速器的变体，缩小延迟差异。
- 相比传统调度，减少40.58%以上任务超时率，精度损失仅2.24%。
- 适合对延迟敏感、需跨异构硬件部署的实时多模型推理场景。
异构深度神经网络（DNN）加速器通过将各层映射到最适配的加速器来降低延迟，实现软实时多模型执行。然而，在负载不均情况下，不同加速器间层延迟差异显著，限制了调度灵活性并增加截止时间超时。为此，本文提出层变体概念，即针对非首选加速器定制化层实现，以缩小延迟差距。进而提出Terastal框架，结合离线异构感知虚拟预算分配与层变体设计，以及在线调度，联合优化加速器映射与变体选择，在满足时序和精度约束下实现最优性能。实验表明，相比FCFS、EDF和DREAM，Terastal在单模型上分别将截止时间超时率降低40.58%、30.53%和36.27%，且跨模型平均精度损失仅为2.24%。
原文摘要 · Abstract (English)
Heterogeneous DNN accelerators improve soft real-time multi-DNN execution by mapping each layer to its preferred accelerator to reduce latency. However, under skewed workloads, large layer-latency differences across accelerators limit scheduling flexibility and increase deadline misses. To address this challenge, we introduce layer variants, customized layer implementations that reduce latency gaps on non-preferred accelerators. We then present Terastal, a soft real-time framework for layer-variant design and scheduling on heterogeneous DNN accelerators. Terastal combines offline heterogeneity-aware virtual budget assignment and layer-variant design, and online scheduling to jointly optimize accelerator mapping and variant selection under timing and accuracy constraints. Experimental results show that Terastal reduces deadline miss rate per model by 40.58%, 30.53%, and 36.27% compared with FCFS, EDF, and DREAM, respectively, while incurring only 2.24% average normalized accuracy loss across models with variants.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。