用蒙特卡洛树搜索实现高效自动防御,减少样本消耗。
ACDZero: MCTS Agent for Mastering Automated Cyber Defense
- 基于MCTS与图神经网络的规划式防御策略
- 在复杂网络中提升防御奖励与鲁棒性,优于主流强化学习方法
- 适合研究高效自动化攻防系统的开发者
自动化网络防御(ACD)旨在以最少的人工干预保护计算机网络,通过隔离主机、重置服务、部署诱饵或更新访问控制等操作应对入侵。然而,现有基于深度强化学习(RL)的方法在具有巨大状态/动作空间的复杂网络中面临难以探索的问题,需大量样本训练。受此启发,我们将CAGE Challenge 4(CAGE-4 / CC4)中的ACD问题建模为上下文相关的部分可观测马尔可夫决策过程,并提出一种以蒙特卡洛树搜索(MCTS)为核心的规划型防御策略。该方法显式建模探索与利用的权衡,利用统计采样指导探索与决策。我们创新性地使用图神经网络(GNN)将网络观测编码为带属性的图,实现对主机及其关系的置换不变推理。为在复杂搜索空间中保持实用性,我们结合学习到的图嵌入与图编辑动作先验,融合无模型泛化、策略蒸馏与前瞻规划。我们在涵盖多种网络结构和攻击行为的CC4场景中评估该代理,结果表明,基于搜索引导与图嵌入的规划策略,在防御奖励和鲁棒性上均优于当前最优的强化学习基线。
原文摘要 · Abstract (English)
Automated cyber defense (ACD) seeks to protect computer networks with minimal or no human intervention, reacting to intrusions by taking corrective actions such as isolating hosts, resetting services, deploying decoys, or updating access controls. However, existing approaches for ACD, such as deep reinforcement learning (RL), often face difficult exploration in complex networks with large decision/state spaces and thus require an expensive amount of samples. Inspired by the need to learn sample-efficient defense policies, we frame ACD in CAGE Challenge 4 (CAGE-4 / CC4) as a context-based partially observable Markov decision problem and propose a planning-centric defense policy based on Monte Carlo Tree Search (MCTS). It explicitly models the exploration-exploitation tradeoff in ACD and uses statistical sampling to guide exploration and decision making. We make novel use of graph neural networks (GNNs) to embed observations from the network as attributed graphs, to enable permutation-invariant reasoning over hosts and their relationships. To make our solution practical in complex search spaces, we guide MCTS with learned graph embeddings and priors over graph-edit actions, combining model-free generalization and policy distillation with look-ahead planning. We evaluate the resulting agent on CC4 scenarios involving diverse network structures and adversary behaviors, and show that our search-guided, graph-embedding-based planning improves defense reward and robustness relative to state-of-the-art RL baselines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。