发现强化学习网络控制器的最坏情况并实时防护,无需重新训练。
Worst-Case Discovery and Runtime Protection for RL-Based Network Controllers

- 通过双层后悔最大化发现最坏场景,给出性能差距下界。
- 在三种控制器上发现性能下降43%至64%,并用轻量规则修复85%以上差距。
- 适用于需高可靠性的网络控制场景,如流媒体自适应传输。
基于强化学习的控制器在拥塞控制和自适应码率流媒体等网络任务中表现出色,但在某些仍可实现高性能的网络条件下,其性能可能严重下降。由于枚举不可行,且强化学习控制器具有序列性与闭环特性,传统形式化验证方法难以应用。本文提出ReGuard框架,可在不重训的情况下,在推理时发现给定强化学习控制器的最坏场景并加以保护。发现过程被建模为双层后悔最大化问题,提供最坏性能差距的严格下界。所发现的轨迹作为反事实分析,转化为轻量逻辑规则,在检测到风险状态时介入,其余时间保持原行为不变。我们在Pensieve、Sage和Park三个强化学习网络控制器上评估ReGuard,发现其最坏情况下性能比理想值低43%至64%。相比最强基线,ReGuard发现的性能差距大57%至6倍,并通过轻量规则保护将差距缩小79%至85%,同时保持正常性能。其保护效果还扩展至未发现的场景,提升多种网络条件下的整体表现。
原文摘要 · Abstract (English)
RL-based controllers achieve strong average-case performance in networking tasks such as congestion control and adaptive bitrate streaming. Yet their performance can degrade severely under network conditions where strong performance is still achievable. Identifying such conditions and quantifying the resulting performance gap is intractable by enumeration, while the sequential and closed-loop nature of RL controllers makes formal verification methods impractical. We present ReGuard, a framework that discovers worst-case scenarios for a given RL controller and protects it against them at inference time without retraining. Discovery is formulated as a bilevel regret-maximization problem, which yields a certified lower bound on the worst-case performance gap. The discovered trajectories are then analyzed as counterfactuals and compiled into lightweight logic rules that intervene only when a risky state is detected, leaving the controller's behavior unchanged otherwise. We evaluate ReGuard across three RL-based network controllers: Pensieve, Sage, and Park. ReGuard discovers scenarios in which the controller's performance is 43$-$64% worse than what is achievable. ReGuard not only discovers gaps 57% to 6$\times$ larger than those found by the strongest baselines but also shrinks them by 79$-$85% via lightweight rule-based protection while preserving nominal performance. ReGuard's protection extends beyond the scenarios it discovers, improving performance across a wider range of network conditions.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。