用离线强化学习提升实时通信的带宽估计精度与稳定性
Robust Bandwidth Estimation for Real-Time Communication with Offline Reinforcement Learning
- 结合Q-集成与高斯混合策略，降低分布外动作风险
- 相比传统方法，带宽过估误差降18%，用户体验提升18.6%
- 内置回退机制，适合生产环境部署，兼顾效果与安全
准确的带宽估计（BWE）对实时通信（RTC）系统至关重要。传统启发式方法在动态网络中适应性有限，而在线强化学习存在探索成本高和可能引发服务中断的问题。离线强化学习利用真实环境收集的高质量数据，成为有前景的替代方案。然而，仍面临分布外（OOD）动作、从行为多样的数据集中提取策略以及在生产系统中可靠部署等挑战。本文提出RBWE框架，基于离线强化学习，融合Q-集成（多个Q函数的集合）与高斯混合策略，以缓解分布外风险并增强策略学习。通过引入回退机制，在高不确定性时切换至启发式方法，确保部署稳定性。实验表明，RBWE将过估误差降低18%，10%分位数用户体验（QoE）提升18.6%，在真实RTC应用中表现出显著实用性。代码已公开：https://github.com/jiu2021/RBWE_offline。
原文摘要 · Abstract (English)
Accurate bandwidth estimation (BWE) is critical for real-time communication (RTC) systems. Traditional heuristic approaches offer limited adaptability under dynamic networks, while online reinforcement learning (RL) suffers from high exploration costs and potential service disruptions. Offline RL, which leverages high-quality data collected from real-world environments, offers a promising alternative. However, challenges such as out-of-distribution (OOD) actions, policy extraction from behaviorally diverse datasets, and reliable deployment in production systems remain unsolved. We propose RBWE, a robust bandwidth estimation framework based on offline RL that integrates Q-ensemble (an ensemble of Q-functions) with a Gaussian mixture policy to mitigate OOD risks and enhance policy learning. A fallback mechanism ensures deployment stability by switching to heuristic methods under high uncertainty. Experimental results show that RBWE reduces overestimation errors by 18% and improves the 10th percentile Quality of Experience (QoE) by 18.6%, demonstrating its practical effectiveness in real-world RTC applications. The implementation is publicly available at https://github.com/jiu2021/RBWE_offline.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。