构建首个基于大模型生成的无人机智能系统评测基准
UAVBench: An Open Benchmark Dataset for Autonomous and Agentic AI UAV Systems via LLM-Generated Flight Scenarios
- 用分类引导的LLM生成5万条真实飞行场景，结构化编码任务与风险
- 设计5万道多选题覆盖导航、协同等10类认知与伦理推理能力
- 首次在物理真实场景下评估大模型决策能力，适合无人机智能研究者
自主飞行系统越来越多依赖大语言模型（LLMs）进行任务规划、感知和决策，但缺乏标准化且具有物理真实性的评测基准，限制了对其推理能力的系统性评估。为填补这一空白，我们提出UAVBench，一个包含50,000条经验证的无人机飞行场景的开源基准数据集，通过分类引导的LLM提示生成并经多阶段安全验证。每个场景以结构化JSON格式编码，包含任务目标、飞行器配置、环境条件及量化风险标签，统一表征跨领域无人机操作。在此基础上，我们构建UAVBench_MCQ，一个面向推理的扩展版本，包含50,000道多选题，覆盖从空气动力学到多智能体协同等十种认知与伦理推理风格。该框架支持可解释、机器可检的无人机特定认知评估。我们评估了32个顶尖大模型，包括GPT-5、ChatGPT-4o、Gemini 2.5 Flash、DeepSeek V3、Qwen3 235B、ERNIE 4.5 300B，发现其在感知与策略推理上表现良好，但在伦理敏感与资源受限决策上仍存挑战。UAVBench为自主空中系统中代理型AI的评测提供了可复现且物理真实的基线，推动下一代无人机推理智能发展。为支持开放科学，所有数据集、评测脚本及相关材料已发布于GitHub：https://github.com/maferrag/UAVBench
原文摘要 · Abstract (English)
Autonomous aerial systems increasingly rely on large language models (LLMs) for mission planning, perception, and decision-making, yet the lack of standardized and physically grounded benchmarks limits systematic evaluation of their reasoning capabilities. To address this gap, we introduce UAVBench, an open benchmark dataset comprising 50,000 validated UAV flight scenarios generated through taxonomy-guided LLM prompting and multi-stage safety validation. Each scenario is encoded in a structured JSON schema that includes mission objectives, vehicle configuration, environmental conditions, and quantitative risk labels, providing a unified representation of UAV operations across diverse domains. Building on this foundation, we present UAVBench_MCQ, a reasoning-oriented extension containing 50,000 multiple-choice questions spanning ten cognitive and ethical reasoning styles, ranging from aerodynamics and navigation to multi-agent coordination and integrated reasoning. This framework enables interpretable and machine-checkable assessment of UAV-specific cognition under realistic operational contexts. We evaluate 32 state-of-the-art LLMs, including GPT-5, ChatGPT-4o, Gemini 2.5 Flash, DeepSeek V3, Qwen3 235B, and ERNIE 4.5 300B, and find strong performance in perception and policy reasoning but persistent challenges in ethics-aware and resource-constrained decision-making. UAVBench establishes a reproducible and physically grounded foundation for benchmarking agentic AI in autonomous aerial systems and advancing next-generation UAV reasoning intelligence. To support open science and reproducibility, we release the UAVBench dataset, the UAVBench_MCQ benchmark, evaluation scripts, and all related materials on GitHub at https://github.com/maferrag/UAVBench
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。