用GRU与迁移学习实现边缘流处理的主动弹性伸缩,提升资源效率。
Towards a Proactive Autoscaling Framework for Data Stream Processing at the Edge using GRU and Transfer Learning
- 用GRU预测上游负载,结合真实与合成数据集。
- 迁移学习缓解在线分布偏移,降低预测误差至1.3% SMAPE。
- 动态调整算子并行度,在资源受限下保持性能。
随着数字经济产生海量数据,高速处理需求日益关键。边缘计算与数据流处理(DSP)结合,将资源靠近数据源,实时分析高速流动的无界数据。但边缘流处理面临负载波动剧烈的问题,资源分配不足导致瓶颈,过剩则造成浪费。现有反应式方法(如阈值策略、排队论)仅在性能下降后响应,可能违反SLA。尽管强化学习可主动适应,但需大量仿真训练;而预测模型又受在线分布漂移影响,准确率下降。本文提出三步方案:首先,使用GRU神经网络基于真实与合成数据集预测上游负载;其次,通过DTW算法与联合分布适配,将离线训练的预测模型迁移至在线系统,应对域差异;最后,设计水平弹性伸缩模块,根据预测负载动态调整算子并行度,兼顾边缘资源约束。轻量级GRU模型在真实数据集上达到最高1.3% SMAPE,优于CNN、ARIMA与Prophet,且训练时间远低于复杂强化学习模型。
原文摘要 · Abstract (English)
Processing data at high speeds is becoming increasingly critical as digital economies generate enormous data. The current paradigms for timely data processing are edge computing and data stream processing (DSP). Edge computing places resources closer to where data is generated, while stream processing analyzes the unbounded high-speed data in motion. However, edge stream processing faces rapid workload fluctuations, complicating resource provisioning. Inadequate resource allocation leads to bottlenecks, whereas excess allocation results in wastage. Existing reactive methods, such as threshold-based policies and queuing theory scale only after performance degrades, potentially violating SLAs. Although reinforcement learning (RL) offers a proactive approach through agents that learn optimal runtime adaptation policies, it requires extensive simulation. Furthermore, predictive machine learning models face online distribution and concept drift that minimize their accuracy. We propose a three-step solution to the proactive edge stream processing autoscaling problem. Firstly, a GRU neural network forecasts the upstream load using real-world and synthetic DSP datasets. Secondly, a transfer learning framework integrates the predictive model into an online stream processing system using the DTW algorithm and joint distribution adaptation to handle the disparities between offline and online domains. Finally, a horizontal autoscaling module dynamically adjusts the degree of operator parallelism, based on predicted load while considering edge resource constraints. The lightweight GRU model for load predictions recorded up to 1.3\% SMAPE value on a real-world data set. It outperformed CNN, ARIMA, and Prophet on the SMAPE and RMSE evaluation metrics, with lower training time than the computationally intensive RL models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。