arXiv:2604.20811cs.AI2026-04被引 1

测试大模型解析语法的能力,发现其在复杂结构下会丢失语义。

Diagnosing CFG Interpretation in LLMs

论文配图:Diagnosing CFG Interpretation in LLMs
图 1 · 摘自论文原文
  • 用可控压力测试分离语法、行为和语义三方面表现
  • 深度递归和高分支时语义对齐完全失效
  • 模型依赖关键词而非纯符号推理,适合构建语法无关智能体

随着大语言模型越来越多地集成到代理系统中,它们必须遵循动态定义的、机器可读的接口。我们评估了大语言模型作为上下文内解释器的表现:给定一个全新的无上下文语法,大语言模型能否生成语法正确、行为有效且语义忠实的输出?我们提出了RoboGrid框架,通过控制递归深度、表达式复杂度和表面风格等条件,将语法、行为与语义进行解耦测试。实验表明存在一致的层级退化现象:大语言模型通常能保持表面语法,但难以维持结构化语义。尽管思维链(CoT)推理提供部分缓解,但在结构密度高时性能急剧下降,特别是深层递归和高分支情况下,语义对齐完全消失。此外,使用‘外星’词汇的测试显示,大语言模型依赖关键词进行语义自举,而非纯粹的符号归纳。这些发现揭示了实现可靠、语法无关代理所必需的层次状态追踪能力的关键缺口。

原文摘要 · Abstract (English)

As LLMs are increasingly integrated into agentic systems, they must adhere to dynamically defined, machine-interpretable interfaces. We evaluate LLMs as in-context interpreters: given a novel context-free grammar, can LLMs generate syntactically valid, behaviorally functional, and semantically faithful outputs? We introduce RoboGrid, a framework that disentangles syntax, behavior, and semantics through controlled stress-tests of recursion depth, expression complexity, and surface styles. Our experiments reveal a consistent hierarchical degradation: LLMs often maintain surface syntax but fail to preserve structural semantics. Despite the partial mitigation provided by CoT reasoning, performance collapses under structural density, specifically deep recursion and high branching, with semantic alignment vanishing at extreme depths. Furthermore, "Alien" lexicons reveal that LLMs rely on semantic bootstrapping from keywords rather than pure symbolic induction. These findings pinpoint critical gaps in hierarchical state-tracking required for reliable, grammar-agnostic agents.

大模型推理语法理解智能体语义对齐

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。