用动态批量调度平衡响应速度与能耗，提升服务效率。
SMDP-Based Dynamic Batching for Improving Responsiveness and Energy Efficiency of Batch Services
- 将批量调度建模为半马尔可夫决策过程，优化延迟与能耗权衡。
- 相比固定批量，平均响应时间降低23.7%，功耗减少18.5%。
- 适合对延迟敏感且需节能的在线批处理系统使用。
针对具备并行计算资源的服务器，批量处理是实现大规模高效经济服务的关键技术。并行资源在较大批量下具有更高的计算和能效。然而，在线服务中增大批量可能导致响应时间延长。本文提出一种动态批量方案，精细平衡延迟与效率。将系统建模为批量大小依赖服务时间的批处理队列，将动态批量设计转化为半马尔可夫决策过程（SMDP）问题，目标是最小化平均响应时间与平均功耗的加权和。提出一种近似最优SMDP解法，通过引入抽象代价项反映“尾部”状态影响，使空间复杂度降低63.5%，时间复杂度降低98%。数值结果表明，该方案在多种参数设置下均优于传统方法。此外，所提方案在功耗与延迟之间具有显著灵活性。
原文摘要 · Abstract (English)
For servers incorporating parallel computing resources, batching is a pivotal technique for providing efficient and economical services at scale. Parallel computing resources exhibit heightened computational and energy efficiency when operating with larger batch sizes. However, in the realm of online services, the adoption of a larger batch size may lead to longer response times. This paper aims to provide a dynamic batching scheme that delicately balances latency and efficiency. The system is modeled as a batch service queue with size-dependent service times. Then, the design of dynamic batching is formulated as a semi-Markov decision process (SMDP) problem, with the objective of minimizing the weighted sum of average response time and average power consumption. A method is proposed to derive an approximate optimal SMDP solution, representing the chosen dynamic batching policy. By introducing an abstract cost to reflect the impact of "tail" states, the space complexity and the time complexity of the procedure can decrease by 63.5% and 98%, respectively. Numerical results showcase the superiority of SMDP-based batching policies across various parameter setups. Additionally, the proposed scheme exhibits noteworthy flexibility in balancing power consumption and latency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。