构建真实法规场景下的规则推理测试集,评估大模型在复杂政策中的合规判断能力。
DeonticBench: A Benchmark for Reasoning over Rules
- 设计跨四大领域6232个任务,涵盖税法、航旅、移民与住房政策
- 顶尖模型在硬子集上准确率仅44.4%(SARA Numeric)和46.6%(Housing)
- 支持自然语言推理与符号计算双路径,可生成可执行的Prolog程序
大语言模型在处理复杂、上下文相关的规则时仍面临挑战。在法律与政策场景中,这体现为规范性推理:依据明文规则判断义务、许可与禁止。现有基准多聚焦短上下文数学推理,缺乏对长上下文、高风险规范性推理的关注。为此,我们提出DEONTICBENCH,包含6,232个任务,覆盖美国联邦税法、航空行李政策、美国移民管理及州级住房法。这些任务可通过自然语言推理或符号计算两种方式解决。除自由格式思维链外,还支持可选的求解器工作流:模型将法规与案情转化为可执行的Prolog代码,实现形式化问题建模与明确程序追踪。所有实例均提供参考Prolog程序。在前沿大模型与编码模型上,最佳硬子集表现仅为SARA Numeric的44.4%与住房领域的46.6%宏平均F1。我们进一步研究了基于监督微调与强化学习的符号程序生成训练。尽管训练提升了Prolog生成质量,当前强化学习方法仍无法可靠求解这些任务。总体而言,DEONTICBENCH为真实世界领域中的上下文依赖规则推理提供了符号与非符号并行的评估基准。
原文摘要 · Abstract (English)
Reasoning with complex, context-specific rules remains challenging for large language models (LLMs). In legal and policy settings, this manifests as deontic reasoning: reasoning about obligations, permissions, and prohibitions under explicit rules. While many recent benchmarks emphasize short-context mathematical reasoning, fewer focus on long-context, high-stakes deontic reasoning. To address this gap, we introduce DEONTICBENCH, a benchmark of 6,232 tasks across U.S. federal taxes, airline baggage policies, U.S. immigration administration, and U.S. state housing law. These tasks can be approached in multiple ways, including direct reasoning in language or with the aid of symbolic computation. Besides free-form chain-of-thought reasoning, DEONTICBENCH enables an optional solver-based workflow in which models translate statutes and case facts into executable Prolog, leading to formal problem interpretations and an explicit program trace. We release reference Prolog programs for all instances. Across frontier LLMs and coding models, best hard-subset performance reaches only 44.4% on SARA Numeric and 46.6 macro-F1 on Housing. We further study training with supervised fine-tuning and reinforcement learning for symbolic program generation. Although training improves Prolog generation quality, current RL methods still fail to solve these tasks reliably. Overall, DEONTICBENCH provides a benchmark for studying context-grounded rule reasoning in real-world domains under both symbolic and non-symbolic settings.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。