arXiv:2604.21191cs.CLcs.FL2026-04ACL被引 1

将前缀解析转化为普通解析,高效计算语法生成概率。

Prefix Parsing is Just Parsing

论文配图:Prefix Parsing is Just Parsing
图 1 · 摘自论文原文
  • 通过构造新语法实现前缀解析到普通解析的转化
  • 可直接使用现有解析器,开销仅小幅增加
  • 支持快速预测下一个词的概率,适合语言模型应用

前缀解析判断输入前缀能否扩展为给定文法生成的完整字符串。在加权情形下,它还提供前缀概率,这对上下文无关语言建模、心理语言学分析及大语言模型的句法约束生成至关重要。本文提出前缀文法变换,将前缀解析高效地还原为普通解析。给定一个文法,该方法构造出一个生成原字符串所有前缀的新文法。随后,只需在变换后的文法上应用任意标准解析算法即可完成前缀解析,无需修改。该转换简洁且实用:变换后文法仅比原文法略大,且可直接复用所有优化实现,无需开发专用前缀解析算法。此外,本文还提出基于算法微分的方法,用于高效计算下一个词的权重向量(即所有单令牌扩展的前缀权重),从而实现高效的下一个词预测。上述贡献共同构成一个简单、通用且高效的前缀解析框架。

原文摘要 · Abstract (English)

Prefix parsing asks whether an input prefix can be extended to a complete string generated by a given grammar. In the weighted setting, it also provides prefix probabilities, which are central to context-free language modeling, psycholinguistic analysis, and syntactically constrained generation from large language models. We introduce the prefix grammar transformation, an efficient reduction of prefix parsing to ordinary parsing. Given a grammar, our method constructs another grammar that generates exactly the prefixes of its original strings. Prefix parsing is then solved by applying any ordinary parsing algorithm on the transformed grammar without modification. The reduction is both elegant and practical: the transformed grammar is only a small factor larger than the input, and any optimized implementation can be used directly, eliminating the need for bespoke prefix-parsing algorithms. We also present a strategy-based on algorithmic differentiation-for computing the next-token weight vector, i.e., the prefix weights of all one-token extensions, enabling efficient prediction of the next token. Together, these contributions yield a simple, general, and efficient framework for prefix parsing.

语法解析语言模型概率推断

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。