用符号推理提升大模型代理的安全审查,实现数学级保障。
FormalJudge: A Neuro-Symbolic Paradigm for Agentic Oversight
- 结合大模型与形式化验证,自上而下生成可验证的约束条件。
- 在三类基准上平均比传统方法提升16.6%安全检测准确率。
- 适合需要高可靠性安全审核的复杂智能体应用场景。
随着基于大语言模型的智能体在高风险领域应用增多,行为安全性愈发关键。当前主流的‘大模型作为评判者’范式面临根本矛盾:概率系统如何可靠监督另一概率系统而不继承其缺陷?本文提出,形式化验证可从根本上破解此困境,但其应用受限于自然语言需求到形式规范的转化瓶颈。为此,我们提出一种神经符号框架,采用双向形式化思维架构:大模型作为规范编译器,自上而下将人类意图分解为原子可验证约束;再自下而上利用Dafny规范与Z3 SMT求解器证明合规性,获得数学保证而非概率评分。在涵盖行为安全、多领域约束遵守及智能体向上欺骗检测的三个基准上验证,7个智能体模型实验显示,该方法相较‘大模型作为评判者’基线平均提升16.6%,实现弱到强的泛化能力(7B判别器对72B代理欺骗检测准确率超90%),并通过迭代优化实现近线性的安全提升。
原文摘要 · Abstract (English)
As LLM-based agents increasingly operate in high-stakes domains with real-world consequences, ensuring their behavioral safety becomes paramount. The dominant oversight paradigm, LLM-as-a-Judge, faces a fundamental dilemma: how can probabilistic systems reliably supervise other probabilistic systems without inheriting their failure modes? We argue that formal verification offers a principled escape from this dilemma, yet its adoption has been hindered by a critical bottleneck: the translation from natural language requirements to formal specifications. This paper bridges this gap by proposing , a neuro-symbolic framework that employs a bidirectional Formal-of-Thought architecture: LLMs serve as specification compilers that top-down decompose high-level human intent into atomic, verifiable constraints, then bottom-up prove compliance using Dafny specifications and Z3 Satisfiability modulo theories solving, which produces mathematical guarantees rather than probabilistic scores. We validate across three benchmarks spanning behavioral safety, multi-domain constraint adherence, and agentic upward deception detection. Experiments on 7 agent models demonstrate that achieves an average improvement of 16.6% over LLM-as-a-Judge baselines, enables weak-to-strong generalization where a 7B judge achieves over 90% accuracy detecting deception from 72B agents, and provides near-linear safety improvement through iterative refinement.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。