arXiv:2509.17455cs.CLcs.AI2025-09Transactions of th…被引 1

用可执行代码揭示评测文本中的隐含条件,让模型理解更透明可信。

Understanding Benchmark Language Under Weakened Formal Semantics

  • 提取可执行代码作为语义证据,通过运行结果验证理解
  • 在数学、法律、生物等多个领域超越纯文本推理效果
  • 适合需要可解释性与可验证性的复杂任务研究者

当前最先进的NLP评测依赖对自然语言中条件、流程和例外的解读,常需隐含假设与外部知识。构建具有证明论保障的完整语义表示在大规模下不切实际,纯文本推理也难以审视。本文探讨在弱化形式语义保证下,对评测语言的理解能达到何种程度。我们提出‘可计算性’(computables):可执行表示,其运行行为提供语义充分性的操作证据,包括可执行性、执行轨迹和运行失败。通过从外部知识检索并迭代优化可计算性,我们在数学推理、多步推理、因果推断,以及规则和例外密集的法律与生物医学评测中均发现该方法显著优于纯文本推理与单次代码执行。分析表明,这些可计算性提供了可扩展、可检查的语义证据:将评测语言中的条件与例外转化为可执行形式,为证明导向语义与纯文本推理之间搭建了实用桥梁。

原文摘要 · Abstract (English)

State-of-the-art NLP benchmarks require interpretation of natural language that specifies conditions, procedures, and exceptions, often relying on implicit assumptions and external knowledge. Constructing complete semantic representations with proof-theoretic guarantees is frequently impractical at scale, and purely text-based reasoning offers limited means of inspection. This paper asks how much understanding of benchmark language can be achieved when formal semantic guarantees are weakened. We investigate this question by extracting computables: executable representations whose runtime behavior provides operational evidence of semantic adequacy, including executability, execution traces, and runtime failures. We induce and iteratively refine computables for benchmark instances using retrieval from external knowledge. Across mathematical reasoning, multi-step reasoning, causal inference, and rule- and exception-heavy legal and biomedical benchmarks, we find that the proposed approach consistently exceeds text-only reasoning and one-shot code execution. Beyond accuracy, our analyses show that these computables provide scalable, inspectable semantic evidence: they expose conditions and exceptions benchmark language forces into executable form, offering a practical bridge between proof-oriented semantics and purely textual reasoning.

语义理解可解释性可执行性

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。