测试模型在少量证据变化后能否正确修正推理结论。
DeltaLogic: Minimal Premise Edits Reveal Belief-Revision Failures in Logical Reasoning Models
- 通过微小前提修改生成推理修订任务,检验模型信念更新能力。
- 大模型初始推理准确率高,但修订准确率普遍偏低,存在明显惯性。
- 适合关注模型动态推理与稳健性的研究者和开发者。
现有推理评测多基于固定前提集判断答案正确性,却忽视了动态环境中至关重要的信念更新能力。本文提出DeltaLogic,一种将自然语言推理题转化为短时修订事件的基准转换协议。每个事件先基于前提P得出结论,再施加最小编辑δ(P),最后判断原结论是否应保留或修正。基于FOLIO和ProofWriter构建,并对Qwen系列及Phi-4-mini-instruct等小规模因果语言模型进行评估。在30个完整测试项中,Qwen3-1.7B初始准确率达0.667,但修订准确率仅0.467,当正确答案应改变时惯性高达0.600;而Qwen3-0.6B几乎全数放弃回答。尽管Qwen3-4B保持相似惯性模式(初始0.650,修订0.450,惯性0.600),Phi-4-mini-instruct表现更优(初始0.950,修订0.850),但仍存在非零抽象与控制不稳现象。结果表明,固定前提下的逻辑能力并不等同于局部证据变动后的合理信念修正。因此,DeltaLogic聚焦一种独立且实际重要的推理能力,补充现有逻辑推理与信念更新评测。
原文摘要 · Abstract (English)
Reasoning benchmarks typically evaluate whether a model derives the correct answer from a fixed premise set, but they under-measure a closely related capability that matters in dynamic environments: belief revision under minimal evidence change. We introduce DeltaLogic, a benchmark transformation protocol that converts natural-language reasoning examples into short revision episodes. Each episode first asks for an initial conclusion under premises P, then applies a minimal edit δ(P), and finally asks whether the previous conclusion should remain stable or be revised. We instantiate DeltaLogic from FOLIO and ProofWriter and evaluate small causal language models with constrained label scoring. On a completed 30-episode Qwen evaluation subset, stronger initial reasoning still does not imply stronger revision behavior: Qwen3-1.7B reaches 0.667 initial accuracy but only 0.467 revision accuracy, with inertia rising to 0.600 on episodes where the gold label should change, while Qwen3-0.6B collapses into near universal abstention. There, Qwen3-4B preserves the same inertial failure pattern (0.650 initial, 0.450 revised, 0.600 inertia), whereas Phi-4-mini-instruct is substantially stronger (0.950 initial, 0.850 revised) but still exhibits non-trivial abstention and control instability. These results suggest that logical competence under fixed premises does not imply disciplined belief revision after local evidence edits. DeltaLogic therefore targets a distinct and practically important reasoning capability that complements existing logical inference and belief-updating benchmarks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。