让AI推理过程可审计,通过显式证明步骤验证答案可信度。
Theoria: Rewrite-Acceptability Verification over Informal Reasoning States
- 将答案重写为带理由的类型化状态转换序列,每步可独立审查。
- 在185个专家问题上达91.4%严格准确率,对抗样本中检出率高出11.5个百分点。
- 适合需要可解释性与高可信度的学术、安全关键场景。
AI何时值得信赖?形式化证明工具虽可靠却覆盖有限;标量大模型评分虽广但结果不可审计且存在一致性问题。我们提出Theoria,一种验证架构:将候选答案重写为一系列带显式理由(引用、计算或题设事实)的类型化状态转移,每步均可独立审计。核心原则是变更完整性——相邻证明状态间所有差异必须被解释,隐藏前提会暴露为未授权的突变。在HLE-Verified Gold(185个纯文本专家问题)上,Theoria以91.4%的严格准确率认证了105个问题(威尔逊95%置信区间[84.5%, 95.4%])。每个认证均生成人类可读的证明轨迹,每步可单独质疑。整体大模型判别器在匹配覆盖率下表现相当,但误判问题不同(杰卡德相似度0.14–0.36),二者互补。在95个跨15领域的对抗污染证明中,结构化判别器检出率达94.7%,显著高于整体判别器的83.2%(p=0.0017)。11.5个百分点的差距集中于隐藏前提(90.6% vs. 62.5%,差28个百分点)和伪造引用(100% vs. 90%),正是形式分析预测的优势领域;在算术与定理误用错误上两者表现一致,无明显差异。在GPQA Diamond(n=65)上,认证准确率达97.1%(威尔逊置信区间[85.1%, 99.5%])。
原文摘要 · Abstract (English)
When should an AI system's answer be trusted? Formal proof assistants offer certainty but cannot reach most of the problem distribution; scalar LLM judges offer coverage but produce opaque scores that cannot be audited after the fact and are subject to the same coherence issues as any LLM. We present Theoria, a verification architecture that closes this gap. A candidate solution is rewritten into a sequence of typed state transitions, each licensed by an explicit justification, whether that be a citation, computation, or problem-given fact, and every transition is independently auditable. The foundational invariant is completeness of change: every difference between consecutive proof states must be accounted for, so hidden premises surface as unlicensed mutations rather than passing silently. On HLE-Verified Gold (185 text-only expert problems), Theoria certifies 105 at 91.4% strict precision (Wilson 95% CI [84.5%, 95.4%]). Every certification produces a human readable proof trace in which each step can be independently challenged. Holistic LLM judges achieve comparable precision at matched coverage but fail on different problems (Jaccard 0.14-0.36), making the approaches complementary. On 95 adversarial poisoned proofs across 15 domains, structured judges catch 94.7% versus 83.2% for holistic judging (p= 0.0017). The overall 11.5 pp gap concentrates in hidden premises (90.6% vs. 62.5%, a 28 pp difference) and fabricated citations (100% vs. 90%), the error classes where the formal analysis predicts an advantage; performance is identical on arithmetic and theorem-misapplication errors, where no advantage is predicted. On GPQA Diamond (n= 65), certified precision is 97.1% (Wilson CI [85.1%, 99.5%]).
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。