用符号化属性分析DRL在系统中的行为,发现更全面的潜在问题。
Analyzing Symbolic Properties for DRL Agents in Systems and Networking
- 提出符号化属性框架,通过单调性和鲁棒性检验跨状态行为
- 实测显示符号属性覆盖范围比传统方法广得多,能发现隐藏缺陷
- 适合关注DRL部署安全性的系统工程师和验证研究者
深度强化学习(DRL)在自适应视频流、无线资源管理与拥塞控制等系统任务中表现优异。但为保障安全部署,需分析代理在真实场景中各类状态下的行为。现有基于验证的方法多聚焦于固定输入点的性质,覆盖有限且需大量人工标注。本文研究系统与网络中DRL代理的符号化属性,提出通用形式化框架,以单调性与鲁棒性为例,并利用现有DNN验证引擎进行分析。方法将符号属性编码为同一策略在相关执行间的比较,分解为可计算的子性质。基于该框架构建diffRL工具,对三个基于DRL的控制系统进行广泛实证研究。结果表明,符号属性显著提升分析覆盖范围,可发现非显性的操作级反例;同时揭示模型规模对可验证性的影响及不同验证后端的实用权衡。
原文摘要 · Abstract (English)
Deep reinforcement learning (DRL) has shown remarkable performance on complex control problems in systems and networking, including adaptive video streaming, wireless resource management, and congestion control. For safe deployment, however, it is critical to reason about how agents behave across the range of system states they encounter in practice. Existing verification-based methods in this domain primarily focus on point properties, defined around fixed input states, which offer limited coverage and require substantial manual effort to identify relevant input-output pairs for analysis. In this paper, we study symbolic properties, that specify expected behavior over ranges of input states, for DRL agents in systems and networking. We present a generic formulation for symbolic properties, with monotonicity and robustness as concrete examples, and show how they can be analyzed using existing DNN verification engines. Our approach encodes symbolic properties as comparisons between related executions of the same policy and decomposes them into practically tractable sub-properties. These techniques serve as practical enablers for applying existing verification tools to symbolic analysis. Using our framework, diffRL, we conduct an extensive empirical study across three DRL-based control systems, adaptive video streaming, wireless resource management, and congestion control. Through these case studies, we analyze symbolic properties over broad input ranges, examine how property satisfaction evolves during training, study the impact of model size on verifiability, and compare multiple verification backends. Our results show that symbolic properties provide substantially broader coverage than point properties and can uncover non-obvious, operationally meaningful counterexamples, while also revealing practical solver trade-offs and limitations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。