用强化学习+安全约束实现网络负载均衡,训练测试都保安全且性能接近最优。
Safe Load Balancing in Software-Defined-Networking

- 在DRL基础上加控制屏障函数,确保决策过程安全
- 测试中端到端延迟表现接近最优,且链路容量不超限
- 预训练模型可迁移至更真实模拟器,微调后效果更好
高性能、高可靠和安全性是软件定义网络(SDN)系统的关键特性。尽管深度强化学习(DRL)被广泛研究以提升性能,但其实际应用受限于探索与决策过程中无法保证安全。为此,本文在DRL上引入控制屏障函数(CBF),设计用于负载均衡的安全控制机制。实验表明,DRL-CBF方法在训练与测试阶段均满足安全要求,同时测试性能接近最优。我们使用两种仿真器验证:基于流的仿真器用于概念验证与基准对比,通过SCIP求解器求解非线性规划(NLP)问题获得最优策略;基于包的仿真器实现真实协议与调度,更贴近实际。结果显示,基于流的仿真器中预训练模型经少量微调即可迁移到包仿真器,性能相近。整体上,在保障链路容量约束的前提下,实现了接近最优的端到端延迟表现。在包仿真器中,本方法优于传统非强化学习基线算法。微调数个训练回合后,训练过程更平稳,测试性能与从零训练相当。
原文摘要 · Abstract (English)
High performance, reliability and safety are crucial properties of any Software-Defined-Networking (SDN) system. Although the use of Deep Reinforcement Learning (DRL) algorithms has been widely studied to improve performance, their practical applications are still limited as they fail to ensure safe operations in exploration and decision-making. To fill this gap, we explore the design of a Control Barrier Function (CBF) on top of Deep Reinforcement Learning (DRL) algorithms for load-balancing. We show that our DRL-CBF approach is capable of meeting safety requirements during training and testing while achieving near-optimal performance in testing. We provide results using two simulators: a flow-based simulator, which is used for proof-of-concept and benchmarking, and a packet-based simulator that implements real protocols and scheduling. Thanks to the flow-based simulator, we compared the performance against the optimal policy, solving a Non Linear Programming (NLP) problem with the SCIP solver. Furthermore, we showed that pre-trained models in the flow-based simulator, which is faster, can be transferred to the packet simulator, which is slower but more accurate, with some fine-tuning. Overall, the results suggest that near-optimal Quality-of-Service (QoS) performance in terms of end-to-end delay can be achieved while safety requirements related to link capacity constraints are guaranteed. In the packet-based simulator, we also show that our DRL-CBF algorithms outperform non-RL baseline algorithms. When the models are fine-tuned over a few episodes, we achieved smoother QoS and safety in training, and similar performance in testing compared to the case where models have been trained from scratch.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。