构建支持持续动作的多智能体攻防仿真环境,用于训练自适应网络安全防御模型。
NetForge RL: A Multi-Agent Simulation Environment for Cyber Defense with Durative Actions
- 基于程序化生成网络,模拟真实攻防场景,支持多防御者协作与自适应对手。
- 在CPU上实现每秒25万步环境迭代,加速强化学习训练过程。
- 提供可复现的评估框架与六项技能诊断工具,适合安全研究与模型验证。
训练强化学习攻防智能体需要反映真实操作环境的仿真系统:观测噪声大、信息不完整、多个防守方跨网络协同,以及通过自对弈实现的自适应攻击者。我们提出NetForge RL,一个面向企业级和工业控制系统(OT)网络的多智能体仿真环境。红方智能体在部分可观测条件下渗透主机;三个分区域蓝方智能体基于合成的SIEM遥测数据与编码后的Windows/Sysmon事件日志进行防御,而非依赖真值状态向量。环境遵循PettingZoo并行API,采用固定形状观测与映射至MITRE ATT&CK的动作空间,包含五个场景及命名难度预设,并提供保留评估集,可在种子下确定性重放。采用JAX后端向量化简化过渡核心,在批量4096时达到每秒2.5×10⁵环境步,作为训练循环的快速替代。配套提供基准模型(脚本化策略、JAX IPPO训练器、自对弈锦标赛)、六项诊断探针(分别衡量一项防御能力)以及报告95%置信区间的评估运行器。文中详述环境的可复现性工程设计,并附负责任使用声明。
原文摘要 · Abstract (English)
Training reinforcement-learning agents for cyber defense requires an environment that reflects the operational setting: noisy, partial observations, several defenders coordinating across a network, and an adaptive adversary realized through self-play. We present NetForge RL, a multi-agent environment for this setting on procedurally generated enterprise and operational-technology (OT) networks. A red agent compromises hosts with partial observability; three zone-split blue agents defend from synthetic SIEM telemetry, Windows/Sysmon event logs encoded into dense embeddings rather than a ground-truth state vector. The environment follows the PettingZoo parallel API with fixed-shape observations and MITRE ATT&CK-mapped actions, ships five scenarios with named difficulty presets and a held-out evaluation split, and replays deterministically under a seed. A JAX backend vectorizes a reduced transition core, reaching 2.5 x 10^5 environment-steps/s at batch 4096 on CPU, as a fast surrogate for training-loop iteration. Alongside the environment we provide reference baselines (scripted, a JAX IPPO trainer, and a self-play tournament), six diagnostic probes that each measure one defensive skill, and an evaluation runner reporting 95% confidence intervals. We describe the reproducibility engineering behind the environment and include a responsible-use statement.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。