首个面向真实场景的LLM智能体安全评测基准，揭示部署风险
RAS-Eval: A Comprehensive Benchmark for Security Evaluation of LLM Agents in Real-World Environments
- 构建涵盖80个测试用例、3802项攻击任务的安全评测框架
- 实测显示攻击使任务完成率平均下降36.78%，成功率高达85.65%
- 验证模型规模越大安全性越强，适合安全研究与产品验收使用
大型语言模型（LLM）智能体在医疗、金融等关键领域的快速部署亟需可靠的安全部署框架。针对动态环境中缺乏标准化评估基准的问题，我们提出RAS-Eval，一个支持模拟与真实工具执行的综合性安全评测基准。该基准包含80个测试用例和3,802项攻击任务，覆盖11类常见漏洞（CWE），工具实现采用JSON、LangGraph和模型上下文协议（MCP）格式。我们在多样化场景下评估了6个主流LLM，发现攻击导致智能体任务完成率（TCR）平均下降36.78%，在学术环境中成功率达85.65%。值得注意的是，安全能力遵循缩放规律，大模型优于小模型。研究揭示了实际部署中的重大安全隐患，并为未来安全研究提供了基础框架。代码与数据已开源于https://github.com/lanzer-tree/RAS-Eval。
原文摘要 · Abstract (English)
The rapid deployment of Large language model (LLM) agents in critical domains like healthcare and finance necessitates robust security frameworks. To address the absence of standardized evaluation benchmarks for these agents in dynamic environments, we introduce RAS-Eval, a comprehensive security benchmark supporting both simulated and real-world tool execution. RAS-Eval comprises 80 test cases and 3,802 attack tasks mapped to 11 Common Weakness Enumeration (CWE) categories, with tools implemented in JSON, LangGraph, and Model Context Protocol (MCP) formats. We evaluate 6 state-of-the-art LLMs across diverse scenarios, revealing significant vulnerabilities: attacks reduced agent task completion rates (TCR) by 36.78% on average and achieved an 85.65% success rate in academic settings. Notably, scaling laws held for security capabilities, with larger models outperforming smaller counterparts. Our findings expose critical risks in real-world agent deployments and provide a foundational framework for future security research. Code and data are available at https://github.com/lanzer-tree/RAS-Eval.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。