评测大模型在规则驱动场景中的推理能力，发现顶尖模型仅达50%表现。
RuleWeaver: Benchmarking Rule-Centered Scenario Reasoning for Large Language Models
- 从规则语料生成条件规则，构建复杂场景问答任务。
- 11个模型平均表现不足满分一半，最高仅50%评分。
- 支持过程评估，可分析规则召回与使用精度，适合领域模型测试。
大型语言模型（LLMs）在专业化领域应用日益广泛，有效利用领域知识常需对具体场景中的复杂规则进行推理。然而现有基准仅部分评估该能力，或仅关注输出指令约束，或忽略规则在场景推理中的独特作用。为此，本文提出RuleWeaver，一个用于评估规则中心化场景推理的基准构建框架。RuleWeaver从语料库提取的IF-THEN元规则出发，逐步扩展为复杂规则，并将其组合成规则中心的场景问答实例。除最终答案正确性外，还支持基于评分标准的答案质量、规则召回率和规则精确度的过程级评估。在11个代表性大模型上的实验表明，当前模型在复杂规则场景推理中仍表现不佳，即使最佳模型也仅达到约50%的最大评分。代码与数据集已公开：https://github.com/SharkSpicy-NLP/RuleWeaver。
原文摘要 · Abstract (English)
Large language models (LLMs) are increasingly applied to specialized domains, where effective use of domain expertise often requires reasoning over complex rules in concrete scenarios. However, existing benchmarks only partially evaluate this capability, as they either focus on output-level instruction constraints or overlook the distinct roles that rules play in scenario reasoning. To address these gaps, this paper introduces RuleWeaver, a benchmark construction framework for evaluating rule-centered scenario reasoning. RuleWeaver starts from corpus-derived IF-THEN Meta Rules, progressively augments them into complex rules, and composes these rules into rule-centered scenario QA instances. Beyond final-answer correctness, RuleWeaver further supports process-level evaluation through rubric-based answer quality, rule recall, and rule precision. Experiments on 11 representative LLMs show that current models still struggle with complex rule-centered scenario reasoning, with even the best-performing model achieving only around 50% of the maximum rubric score. We make our code and dataset available here: https://github.com/SharkSpicy-NLP/RuleWeaver.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。