用智能体评估逻辑推理模型，确保结果可复现且抗失败。
Agentified Assessment of Logical Reasoning Agents
- 设计评估智能体自动发任务、控资源、记录失败类型。
- 在清洗后的FOLIO数据集上达到86.70%准确率，优于基线73.89%。
- 适合研究逻辑推理、自动化验证的开发者参考。
我们提出一个可复现、可审计且对执行失败鲁棒的逻辑推理智能体评估框架。基于智能体化评估，使用评估智能体发出任务、控制执行预算、解析输出并记录结构化失败类型，被测智能体只需提供标准化的智能体-智能体接口。以一阶逻辑（FOL）推理为例，我们在经过求解器验证与修复的FOLIO数据集上评估了自动形式化智能体。该智能体将自然语言前提和结论转换为可执行的Z3Py程序，并利用满足模理论（SMT）求解判断逻辑蕴含关系。在清理后的FOLIO验证集上，该智能体在评估协议下取得86.70%的准确率，显著高于链式思维基线（73.89%）。
原文摘要 · Abstract (English)
We present a framework for evaluating and benchmarking logical reasoning agents when assessment itself must be reproducible, auditable, and robust to execution failures. Building on agentified assessment, we use an assessor agent to issue tasks, enforce execution budgets, parse outputs, and record structured failure types, while the agent under test only needs to expose a standardized agent-to-agent interface. As a case study, we benchmark an auto-formalization agent for first-order logic (FOL) reasoning on a solver-verified and repaired split of FOLIO. The agent translates natural language premises and conclusions into executable Z3Py programs and employs satisfiability modulo theories (SMT) solving to determine logical entailment. On the cleaned FOLIO validation set, the auto-formalization agent achieves 86.70% accuracy under the assessor protocol, outperforming a chain-of-thought baseline (73.89%).
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。