让大模型生成代码时既保证语法正确，又能自主学习结构信息。
Weave of Formal Thought
- 用强化的解析器和推测性词法分析，确保输出代码始终合法。
- 通过隐变量训练，让模型在生成时主动保留语法结构信息。
- 适合需要高可靠性代码生成的场景，如自动化编程或形式化验证。
大型语言模型（LLMs）在代码生成上表现出色，但无法形式化保证输出的语法正确性，也未能有效利用目标语言的层次结构。现有约束解码框架虽解决部分语法问题，但受限于僵化假设，难以支持上下文敏感词法、最大匹配分词和关键词提取等关键机制，且仅近似词汇掩码，导致不完整。针对结构学习，当前方法多依赖预设策略注入语法信息，而非让模型自主学习应暴露哪些结构。本文提出「形式思维编织」（Weave of Formal Thought, WoFT），统一实现严格的语法验证与可学习的结构表征。首先，设计基于Tree-sitter规范的完备且正确的解析引擎与约束解码器，通过将广义LR（GLR）解析与推测性词法分析结合，同步维护多个词法状态假设，并在图结构栈中推进，确保所有子词扩展均能构成合法程序前缀，拒绝非法扩展。其次，提出一种隐变量微调方法，使语言模型在生成时直接插入非终结符符号。利用重加权唤醒-睡眠（RWS）算法优化重要性加权证据下界（IW-ELBO），模型学会选择性保留形式推导过程，作为自适应的结构草稿。在Python任务上，使用RWS目标对StarCoder2-3B进行微调，相比纯文本监督微调（SFT）基线，每标记交叉熵降低14.3%，表明有选择性的潜在语法信息可恢复传统自回归训练所丢弃的关键结构知识。
原文摘要 · Abstract (English)
Large language models (LLMs) attain remarkable surface fluency on code, yet they neither formally guarantee the syntactic validity of their output nor leverage the hierarchical structure defining the target language. While existing constrained-decoding frameworks address the former, they operate under rigid assumptions that preclude critical lexical mechanisms -- including context-sensitive lexing, maximal-munch tokenization, and keyword extraction -- and only approximate vocabulary masking, sacrificing completeness. For the latter, code LLMs typically inject grammatical structure via predetermined policies rather than learning which structural information to expose. In this work, we introduce Weave of Formal Thought (WoFT), a paradigm uniting rigorous syntactic validation with learned structural representations. First, we present a formal engine and constrained decoder that is sound and complete with respect to the full Tree-sitter specification. By augmenting generalized LR (GLR) parsing with a speculative-lexing construction that maintains concurrent lexer-state hypotheses synchronized with a GLR graph-structured stack, our decoder admits every subword token extending to a valid program prefix and rejects all others. Second, we present a latent-variable fine-tuning method training the language model to interleave non-terminal grammar symbols directly into generation. Utilizing the reweighted wake-sleep (RWS) algorithm to optimize the importance-weighted evidence lower bound (IW-ELBO) of the surface text, the model learns to selectively retain formal derivations as an adaptive structural scratchpad. For Python, fine-tuning StarCoder2-3B with our RWS objective reduces per-token cross-entropy by 14.3% relative to a text-only SFT baseline, demonstrating that discretionary latent syntax recovers critical structural information that flat autoregressive training discards.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。