arXiv:2604.18328cs.CL2026-04ACL被引 2

用混合神经符号方法提升逻辑推理准确率,降低内容偏见影响。

FregeLogic at SemEval 2026 Task 11: A Hybrid Neuro-Symbolic Architecture for Content-Robust Syllogistic Validity Prediction

论文配图:FregeLogic at SemEval 2026 Task 11: A Hybrid Neuro-Symbolic Architecture for Content-Robust Syllogistic Validity Prediction
图 1 · 摘自论文原文
  • 集成多个大模型并用Z3逻辑求解器处理分歧案例
  • 准确率达94.3%,内容干扰降至2.85,综合得分41.88
  • 适合关注逻辑推理鲁棒性的AI系统开发者

我们提出FregeLogic,一种用于SemEval-2026任务11(子任务1)的混合神经符号系统,旨在减少内容因素对逻辑有效性判断的影响。该方法结合五个大语言模型分类器(基于Llama 4 Maverick、Llama 4 Scout、Qwen3-32B三种开源模型及不同提示策略)的集成,与Z3 SMT求解器协同工作,作为形式逻辑校验器。核心假设是:当大模型间出现分歧时,往往源于现实可信度对逻辑判断的干扰。系统在这些争议案例中转而依赖Z3进行结构化形式验证,最终在包含960个样本的数据集上实现94.3%的准确率,内容效应降至2.85,综合得分为41.88(嵌套5折交叉验证)。相比纯集成模型(综合得分39.12),综合得分提升2.76点,准确率提高0.9%,内容效应下降16%(从3.39降至2.85)。通过结构化输出接口调用Z3,失败率从约22%降至接近零;采用带存在公理的亚里士多德编码方式,并经任务标注验证有效。结果表明,在模型共识最低处精准引入形式方法,可有效提升任务所用的准确率加内容效应综合指标。

原文摘要 · Abstract (English)

We present FregeLogic, a hybrid neuro-symbolic system for SemEval-2026 Task 11 (Subtask 1), which addresses syllogistic validity prediction while reducing content effects on predictions. Our approach combines an ensemble of five LLM classifiers, spanning three open-weights models (Llama 4 Maverick, Llama 4 Scout, and Qwen3-32B) paired with varied prompting strategies, with a Z3 SMT solver that serves as a formal logic tiebreaker. The central hypothesis is that LLM disagreement within the ensemble signals likely content-biased errors, where real-world believability interferes with logical judgment. By deferring to Z3's structurally-grounded formal verification on these disputed cases, our system achieves 94.3% accuracy with a content effect of 2.85 and a combined score of 41.88 in nested 5-fold cross-validation on the dataset (N=960). This represents a 2.76-point improvement in combined score over the pure ensemble (39.12), with a 0.9% accuracy gain, driven by a 16% reduction in content effect (3.39 to 2.85). Adopting structured-output API calls for Z3 extraction reduced failure rates from ~22% to near zero, and an Aristotelian encoding with existence axioms was validated against task annotations. Our results suggest that targeted neuro-symbolic integration, applying formal methods precisely where ensemble consensus is lowest, can improve the combined accuracy-plus-content-effect metric used by this task.

逻辑推理神经符号大模型集成

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。