用逻辑规则生成测试用例，评估大模型调用工具时的隐性合规性。
Evaluating Implicit Regulatory Compliance in LLM Tool Invocation via Logic-Guided Synthesis
- 将法规转化为时间逻辑约束，自动合成安全关键测试路径。
- 13个SOTA模型中大模型更易忽视安全而优先完成任务。
- 适合关注AI安全、监管合规的开发者和研究者。
大型语言模型（LLMs）在自主代理中的集成已实现复杂工具调用，但在高风险领域，系统不仅需功能正确，还必须严格遵守监管标准。现有基准普遍忽略隐性合规性，无法评估模型是否能自主执行强制性安全约束。为此，我们提出LogiSafetyGen框架，将非结构化法规转换为线性时序逻辑（LTL）断言，并通过逻辑引导的模糊测试合成有效且具有安全挑战性的执行轨迹。基于该框架，我们构建了LogiSafetyBench基准，包含240个经人工验证的任务，要求模型生成满足功能目标与潜在合规规则的Python程序。对13个前沿大模型的评估显示，尽管更大模型在功能正确性上表现更好，但常因优先完成任务而违反安全约束，导致不合规行为。
原文摘要 · Abstract (English)
The integration of large language models (LLMs) into autonomous agents has enabled complex tool use, yet in high-stakes domains, these systems must strictly adhere to regulatory standards beyond simple functional correctness. However, existing benchmarks often overlook implicit regulatory compliance, thus failing to evaluate whether LLMs can autonomously enforce mandatory safety constraints. To fill this gap, we introduce LogiSafetyGen, a framework that converts unstructured regulations into Linear Temporal Logic oracles and employs logic-guided fuzzing to synthesize valid, safety-critical traces. Building on this framework, we construct LogiSafetyBench, a benchmark comprising 240 human-verified tasks that require LLMs to generate Python programs that satisfy both functional objectives and latent compliance rules. Evaluations of 13 state-of-the-art (SOTA) LLMs reveal that larger models, despite achieving better functional correctness, frequently prioritize task completion over safety, which results in non-compliant behavior.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。