为军事场景设计的LLM安全评测基准，填补了军用伦理规范测试空白。
ARMOR 2025: A Military-Aligned Benchmark for Evaluating Large Language Model Safety Beyond Civilian Contexts
- 基于战争法、交战规则等三大军事教义构建测评题库
- 覆盖12类军事决策场景，测试21款商用大模型表现
- 适合军方、国防科技机构评估AI在作战中的合规性
大型语言模型正被探索用于需可靠且合法合规决策支持的国防应用，有望提升军事决策、协同与作战效率。然而现有安全评测聚焦通用社会风险，未能检验模型是否遵守真实军事行动中的法律与伦理规范。为此，我们提出ARMOR 2025——一个基于战争法、交战规则和联合道德条例三大军事教义的军事对齐安全评测基准。从这些权威文献中提取文本并生成保持原意的多选题，按观察-判断-决策-行动（OODA）框架组织，形成12个类别、519个基于教义的提示，对21款商用大模型进行严格评估。结果揭示了当前模型在军事应用中的安全对齐存在关键缺陷。
原文摘要 · Abstract (English)
Large language models (LLMs) are now being explored for defense applications that require reliable and legally compliant decision support. They also hold significant potential to enhance decision making, coordination, and operational efficiency in military contexts. These uses demand evaluation methods that reflect the doctrinal standards that guide real military operations. Existing safety benchmarks focus on general social risks and do not test whether models follow the legal and ethical rules that govern real military operations. To address this gap, we introduce ARMOR 2025, a military aligned safety benchmark grounded in three core military doctrines the Law of War, the Rules of Engagement, and the Joint Ethics Regulation. We extract doctrinal text from these sources and generate multiple choice questions that preserve the intended meaning of each rule. The benchmark is organized through a taxonomy informed by the Observe Orient Decide Act (OODA) decision making framework. This structure enables systematic testing of accuracy and refusal across military relevant decision types. This benchmark features a structured 12-category taxonomy, 519 doctrinally grounded prompts, and rigorous evaluation procedures applied to 21 commercial LLMs. Evaluation results reveal critical gaps in safety alignment for military applications.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。