首个面向6G下LLM无人机的安全评估基准，测试对抗干扰下的可靠性
$α^3$-SecBench: A Large-Scale Evaluation Suite of Security, Resilience, and Trust for LLM-based UAV Agents over 6G Networks
- 构建2万+真实攻击场景，覆盖7个自主层级的对抗测试
- 23个主流大模型平均得分仅12.9%~57.1%，检测与应对能力严重脱节
- 适合关注AI安全、无人机系统鲁棒性及6G智能应用的研究者
自主无人飞行器（UAV）系统正广泛部署于高安全性网络环境中，需在恶意攻击下保持可靠运行。尽管现有基准已评估基于大语言模型（LLM）的UAV在推理、导航和效率方面的能力，但对安全、韧性与信任在对抗条件下的系统性评估仍属空白，尤其在新兴的6G环境。本文提出$α^{3}$-SecBench，首个针对LLM驱动无人机在真实对抗干扰下安全意识自主性的大规模评测套件。基于$α^{3}$-Bench的多轮对话式任务，该框架在良性任务中嵌入20,000个经验证的攻击场景，覆盖感知、感知、规划、控制、通信、边缘/云基础设施及LLM推理共七个自主层级。评估涵盖三个维度：安全（攻击检测与漏洞归因）、韧性（安全退化行为）与信任（合规工具使用）。我们使用来自主要工业厂商与领先研究机构的23个前沿大模型，在113,475次任务构成的语料库中采样数千条对抗增强的飞行任务进行测试，涵盖175种威胁类型。尽管多数模型能可靠检测异常行为，但有效缓解、漏洞归因与可信控制行为仍不一致。综合得分范围为12.9%至57.1%，凸显了异常检测与安全自主决策之间的显著差距。项目已开源：https://github.com/maferrag/AlphaSecBench
原文摘要 · Abstract (English)
Autonomous unmanned aerial vehicle (UAV) systems are increasingly deployed in safety-critical, networked environments where they must operate reliably in the presence of malicious adversaries. While recent benchmarks have evaluated large language model (LLM)-based UAV agents in reasoning, navigation, and efficiency, systematic assessment of security, resilience, and trust under adversarial conditions remains largely unexplored, particularly in emerging 6G-enabled settings. We introduce $α^{3}$-SecBench, the first large-scale evaluation suite for assessing the security-aware autonomy of LLM-based UAV agents under realistic adversarial interference. Building on multi-turn conversational UAV missions from $α^{3}$-Bench, the framework augments benign episodes with 20,000 validated security overlay attack scenarios targeting seven autonomy layers, including sensing, perception, planning, control, communication, edge/cloud infrastructure, and LLM reasoning. $α^{3}$-SecBench evaluates agents across three orthogonal dimensions: security (attack detection and vulnerability attribution), resilience (safe degradation behavior), and trust (policy-compliant tool usage). We evaluate 23 state-of-the-art LLMs from major industrial providers and leading AI labs using thousands of adversarially augmented UAV episodes sampled from a corpus of 113,475 missions spanning 175 threat types. While many models reliably detect anomalous behavior, effective mitigation, vulnerability attribution, and trustworthy control actions remain inconsistent. Normalized overall scores range from 12.9% to 57.1%, highlighting a significant gap between anomaly detection and security-aware autonomous decision-making. We release $α^{3}$-SecBench on GitHub: https://github.com/maferrag/AlphaSecBench
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。