让深度强化学习在真实网络边缘落地,解决部署中的三大实际难题。
Towards Practical Operation of Deep Reinforcement Learning Agents in Real-World Network Management at Open RAN Edges
- 用MEC+Open RAN架构整合DRL策略,支持多时间尺度运行
- 实测验证三类挑战:突发流量、拓扑异构、训练扰流导致服务中断
- 提出时序融合、多智能体、迁移学习方案,加速上线并保障稳定
深度强化学习(DRL)已成为满足先进网络对连接性、可靠性、低延迟和运营效率日益增长需求的强大解决方案。然而,多数研究集中于理论分析与仿真,缺乏对真实世界部署的探索。为弥合这一差距并推动DRL在网络管理中的实用化,我们首次提出一个集成ETSI多接入边缘计算(MEC)与Open RAN的编排框架,实现DRL策略在不同时间尺度下的无缝应用,并提升智能体生命周期管理能力。随后,我们识别出阻碍DRL真实部署的三个关键挑战:(1)来自不可预测或突发流量的异步请求;(2)在异构拓扑和不断变化的服务需求下适应性和泛化能力不足;(3)在实时运行环境中因探索行为导致收敛时间长和服务中断。针对这些问题,我们提出三项综合解决方案:(a)采用先进的时序数据融合方法处理异步流量;(b)设计灵活架构如多智能体DRL与增量学习以应对异构场景;(c)通过模拟驱动部署结合迁移学习,减少收敛时间与服务中断。最后,在城市级测试基础设施上验证了MEC-O-RAN架构的可行性,并展示了两个真实应用场景,凸显了所识别挑战并证明了所提方案的有效性。
原文摘要 · Abstract (English)
Deep Reinforcement Learning (DRL) has emerged as a powerful solution for meeting the growing demands for connectivity, reliability, low latency and operational efficiency in advanced networks. However, most research has focused on theoretical analysis and simulations, with limited investigation into real-world deployment. To bridge the gap and support practical DRL deployment for network management, we first present an orchestration framework that integrates ETSI Multi-access Edge Computing (MEC) with Open RAN, enabling seamless adoption of DRL-based strategies across different time scales while enhancing agent lifecycle management. We then identify three critical challenges hindering DRL's real-world deployment, including (1) asynchronous requests from unpredictable or bursty traffic, (2) adaptability and generalization across heterogeneous topologies and evolving service demands, and (3) prolonged convergence and service interruptions due to exploration in live operational environments. To address these challenges, we propose a three-fold solution strategy: (a) advanced time-series integration for handling asynchronized traffic, (b) flexible architecture design such as multi-agent DRL and incremental learning to support heterogeneous scenarios, and (c) simulation-driven deployment with transfer learning to reduce convergence time and service disruptions. Lastly, the feasibility of the MEC-O-RAN architecture is validated on an urban-wide testing infrastructure, and two real-world use cases are presented, showcasing the three identified challenges and demonstrating the effectiveness of the proposed solutions.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。