大模型在复杂规则判断中会突然失效，新框架让规则执行确定可控。
Confidently Wrong: Exception Chain Collapse in Frontier LLM Rule Evaluation
- 用神经符号架构让大模型写规则，再由SMT引擎严格执行
- 同一模型在3月到4月间合规率从96.6%升至100%，显示隐性失效
- 适合需要可审计、高可靠规则判断的金融保险等监管场景
我们发现前沿大模型在嵌套条件规则（如“除非B成立，否则需A；除非C覆盖B”）下的资格评估中存在一种故障模式——异常链坍缩。该故障首次出现时可复现，但其表现不稳：2026年3月至4月间，相同模型版本（GPT-5.4）在建筑保险任务上合规率从96.6%无预警提升至100%，提示模型推理边界在无声变化。为此提出Aethis Eligibility Module，结合大模型生成权威规则与SMT引擎的确定性执行，确保结果始终符合规范，不受模型漂移、推理强度或提示格式影响。三重证据：(i) 225个场景跨四领域基准测试验证了该模式及部分修复；(ii) 建筑保险20个对抗性场景中，该引擎全对，仅一个前沿配置（GPT-5.4低推理努力）通过，其余三个包括当时最强的Anthropic模型均失败于覆盖率缺口；(iii) 在LegalBench九项任务共949个未见样本上，该引擎显著优于所有前沿模型（合并McNemar检验p ≤ 0.003），多阶段任务最高领先41分。贡献在于将不确定性从沉默的推理边界转移至可审计的规范边界。所有场景、规则编码与结果公开可复现。
原文摘要 · Abstract (English)
We document a failure class in frontier large language models -- exception chain collapse -- observed in eligibility evaluation under nested conditional rules of the form "A is required UNLESS B applies, UNLESS C overrides B". The failure reproduces at first observation, but its empirical surface is unstable: between March and April 2026 several failure cells closed silently under the same model alias, with no version bump (GPT-5.4 on construction insurance moved from 96.6% to 100%, same prompt and harness). For regulated workflows, frontier-model accuracy is a moving compliance boundary that shifts without notice. We present the Aethis Eligibility Module, a neuro-symbolic architecture in which LLMs author rules from authoritative sources and an SMT-based layer executes them deterministically, consistent with the authored specification regardless of model drift, reasoning-effort defaults, or prompt format. Three evidence bases: (i) a controlled benchmark of 225 scenarios across four regulatory domains documents the pattern and, in replication, the drift that partially closed it; (ii) a 20-scenario adversarial extension on construction insurance, where the engine scores 20/20, as does one of four frontier configurations (GPT-5.4 at low reasoning effort), while the other three, including Anthropic's strongest model at evaluation time, fail the same coverage-gap edge case; (iii) external validation on nine peer-reviewed LegalBench tasks, 949 held-out cases, where the engine is significantly more accurate than all three frontier models (combined McNemar's p <= 0.003), with margins up to +41 points on the curated multi-prong tasks against the Anthropic models. The contribution is to relocate uncertainty from the inference boundary, where it is silent, to the specification boundary, where it is deliberate and audited. All scenarios, rule encodings, and results are public and reproducible.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。