用大模型指导小模型,让轻量级智能体学会高效自主防御。
Distilling Knowledge from Large Language Models into Lightweight Reinforcement Learning Agents for Autonomous Cyber Operations

- 用提示工程调用80亿参数网络安全大模型,生成防御策略。
- 将大模型策略蒸馏到仅6万参数的小模型,性能接近原模型。
- 适合需要快速部署、资源受限的实战化网络安全场景。
自主网络攻防(ACO)在企业网络安全中日益重要,但传统强化学习(RL)代理需大量探索,初期决策不稳定。本文利用一个预训练于网络安全数据的80亿参数大语言模型(LLM),通过提示工程而非微调,在改进的CybORG CAGE Challenge 2环境中表现优于基线RL代理。提出一种在线策略蒸馏框架,将该LLM的防御策略迁移至仅含64,910参数的轻量级RL代理,模型规模缩小数个数量级,同时保持有效防御能力。在包含4至12台主机的多种网络配置下验证了方法可迁移性。评估发现,教师引导的强化学习稳定策略未显著优于优化后的教师策略,表明奖励驱动的RL优化与教师指导策略间存在对齐局限。结果表明,专注网络安全的大模型可作为专家知识源,而策略蒸馏为将前沿模型部署于高效、可扩展的轻量代理提供了可行路径。
原文摘要 · Abstract (English)
Autonomous Cyber Operations (ACO) are increasingly important for defending enterprise networks as cyber threats continue to evolve in sophistication. ACO applications commonly employ Reinforcement Learning (RL) agents to learn defensive behaviors through interaction with environments. However, RL agents typically require extensive exploration during training, often resulting in unstable behavior and poor initial decision-making before converging toward effective defense strategies. In this work, we investigate the use of a Large Language Model (LLM) to improve autonomous defensive decision-making within an ACO environment. Through prompt engineering rather than fine-tuning, we demonstrate that an 8-billion parameter LLM pretrained on cybersecurity data can outperform a baseline RL agent in a modified CybORG CAGE Challenge 2 environment. We then propose an online policy distillation framework that transfers the LLM's defensive policy into a lightweight RL agent containing only 64,910 parameters, reducing model size by several orders of magnitude while maintaining effective defensive capabilities. This provides a pathway toward operationalizing frontier cybersecurity models within lightweight, deployable agents. To evaluate transferability, we construct CybORG scenarios ranging from 4 to 12 hosts and assess the approach across varying network configurations. We also evaluate teacher-guided RL stabilization strategies and observe that none consistently surpass the optimized teacher policy, suggesting policy-alignment limitations between reward-driven RL optimization and teacher-guided defense strategies. Our results demonstrate the potential of cybersecurity-focused LLMs as sources of expertise for autonomous cyber defense, while policy distillation provides a practical path toward operationalizing frontier cybersecurity models within efficient, scalable agents.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。