动态调度模型推理，兼顾延迟、能耗与准确率
Slim Scheduler: A Runtime-Aware RL and Scheduler System for Efficient CNN Inference
- 结合强化学习与贪心算法，分层调度分布式推理任务
- 相比随机分配，平均延迟降96.45%，能耗降97.31%
- 适合对实时性与能效敏感的边缘部署场景
现有神经网络调度研究多聚焦于固定宽度的静态模型，忽视了在异构硬件和动态运行时条件下自适应调度的重要性。本文提出Slim Scheduler，一种融合近端策略优化（PPO）强化学习策略与算法性贪心调度器的混合调度框架，用于协调可剪枝模型的分布式推理。每个服务器运行本地贪心调度器，根据显存与利用率约束批处理兼容请求并管理实例伸缩；同时，PPO路由器学习全局路由策略，包括设备选择、模型宽度比例与批量配置。该分层设计降低搜索空间复杂度，缓解对特定硬件的过拟合，平衡效率与吞吐量。相较于纯随机任务分配基线，Slim Scheduler 在将精度降至最窄模型（70.3%）时，实现平均延迟降低96.45%、能耗降低97.31%；亦可在提升精度的同时，以更高的延迟与能耗标准差为代价，整体降低平均延迟与能耗，提高任务吞吐量。
原文摘要 · Abstract (English)
Most neural network scheduling research focuses on optimizing static, end-to-end models of fixed width, overlooking dynamic approaches that adapt to heterogeneous hardware and fluctuating runtime conditions. We present Slim Scheduler, a hybrid scheduling framework that integrates a Proximal Policy Optimization (PPO) reinforcement learning policy with algorithmic, greedy schedulers to coordinate distributed inference for slimmable models. Each server runs a local greedy scheduler that batches compatible requests and manages instance scaling based on VRAM and utilization constraints, while the PPO router learns global routing policies for device selection, width ratio, and batch configuration. This hierarchical design reduces search space complexity, mitigates overfitting to specific hardware, and balances efficiency and throughput. Compared to a purely randomized task distribution baseline, Slim Scheduler can achieve various accuracy and latency trade-offs such as: A 96.45% reduction in mean latency and a 97.31% reduction in energy usage dropping accuracy to the slimmest model available (70.3%). It can then accomplish an overall reduction in average latency plus energy consumption with an increase in accuracy at the cost of higher standard deviations of said latency and energy, effecting overall task throughput.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。