让语言模型提前预判生成结果，提升可控性与效率。
Learning Tractable Distributions Of Language Model Continuations
- 用神经头预测前缀相关的潜在先验，共享隐马尔可夫模型回答后续问题。
- 在不增加显著延迟的情况下实现100%语法约束满足，解码开销仅增14%。
- 适合需要精准控制生成内容的场景，如安全过滤、风格保持等。
可控生成需施加序列级约束（如语法、风格、安全性），而这些约束依赖未来词元，导致自回归语言模型精确条件化不可行。现有近似方法如隐马尔可夫模型（HMM）虽能逼近延续分布并引导解码，但通常上下文感知能力弱。本文提出「学习提前看」（LTLA），一种混合方法：利用基础语言模型嵌入来条件化全局学习的可计算代理——一个神经头仅预测前缀相关的潜在先验，而共享的HMM则精确回答延续查询。LTLA避免两大效率陷阱：一、通过单次批量HMM前向更新，避免对所有词表候选进行额外语言模型评分（无V次额外评估）；二、不为每个前缀预测新HMM，仅条件化潜在先验，实现解码步骤间缓存的未来似然消息复用。实证表明，相比标准HMM代理，LTLA提升延续概率，支持视觉-语言模型的连续上下文前瞻控制，实现100%语法约束满足，并有效净化内容，解码时间仅增加14%。
原文摘要 · Abstract (English)
Controlled generation imposes sequence-level constraints (syntax, style, safety) that depend on future tokens, making exact conditioning of an autoregressive LM intractable. Tractable surrogates such as HMMs can approximate continuation distributions and steer decoding, but standard surrogates are often weakly context-aware. We propose Learning to Look Ahead (LTLA), a hybrid method that uses base-LM embeddings to condition a globally learned tractable surrogate: a neural head predicts only a prefix-dependent latent prior, while a shared HMM answers continuation queries exactly. LTLA is designed to avoid two common efficiency traps when adding neural context. First, it avoids vocabulary-sized prefix rescoring (V extra LM evaluations) by scoring all next-token candidates via a single batched HMM forward update. Second, it avoids predicting a new HMM per prefix by learning one shared HMM and conditioning only the latent prior, which enables reuse of cached future-likelihood (backward) messages across decoding steps. Empirically, LTLA improves continuation likelihood over standard HMM surrogates, enables lookahead control for vision--language models by incorporating continuous context, achieves 100% syntactic constraint satisfaction, and improves detoxification while adding only a 14% decoding-time overhead.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。