用两阶段训练让小模型在本地完成权限提升,成功率超93%
Towards Reliable Local Security Agents: Verifiable Post-Training for Linux Privilege Escalation
- 分两阶段微调:先监督学习,再用可验证奖励强化学习
- 在20轮内达成93.3%成功率,推理成本降低80倍以上
- 适合需要本地化、高可靠性安全工具的开发者
大型语言模型代理在安全领域日益重要,但主流系统多为闭源、云端部署,难以复现或用于敏感代码。为此,本文提出一种两阶段后训练方法,将小型本地语言模型转化为安全代理。以Linux权限提升为例,该任务具备自动验证性且需多步交互推理。通过避免数据泄露的实验设置,对40亿参数的小模型进行两阶段训练:首先在程序化权限提升环境的轨迹上进行监督微调,随后采用可验证奖励进行强化学习。在包含12个场景的独立测试集上,监督微调使基准成功率翻倍(20轮预算下),后续强化学习使模型PrivEsc-LLM 4B达到93.3%成功,仅略低于Claude Opus 4.7;同时,每次成功提升的预期推理成本下降超过80倍。研究证明小型本地模型可胜任复杂安全任务,并揭示关键挑战,为迁移至其他场景提供指导。
原文摘要 · Abstract (English)
LLM agents are becoming increasingly important in the security domain, but leading systems are often closed-source, cloud-based, hard to reproduce or use with sensitive code. This creates a need for small, local models that can perform security tasks under strict resource constraints, though effective methods for developing them remain unexplored. In this paper, we address this gap by proposing a two-stage post-training recipe that turns a small local language model into a security agent. To this end, we focus on Linux privilege escalation as a representative setting to systematically study the training of local models, as the task is both automatically verifiable and requires multi-step interactive reasoning. Using an experimental setup that mitigates data leakage, we post-train a small 4B model in two stages: supervised fine-tuning on traces from procedural privilege-escalation environments, followed by reinforcement learning with verifiable rewards. On a held-out benchmark of 12 Linux privilege-escalation scenarios, supervised fine-tuning doubles the baseline success rate under a tight budget of 20 interaction rounds, and subsequent reinforcement learning training improves our model, PrivEsc-LLM 4B, to 93.3% success, behind only Claude Opus 4.7 at this budget. At the same time, the expected inference cost per successful escalation decreases by more than 80x. Our findings not only show that small local models can be adapted to complex security tasks, but also document the challenges involved, offering guidance for transferring this recipe to other settings.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。