用对数空间的几何路径实现非自回归文本生成,提升准确率与效率。
Logit-KL Flow Matching: Non-Autoregressive Text Generation via Sampling-Hybrid Inference
- 基于对数空间的KL测地线构建生成路径,理论严谨。
- 在文本和代码补全任务上,困惑度优于现有非自回归模型。
- 提出采样混合策略,兼顾生成质量与推理速度,适合工业应用。
非自回归(NAR)语言模型通过跳过自回归解码的串行瓶颈,显著提升文本生成效率。然而,在离散序列中准确建模依赖关系仍是挑战。本文将条件流匹配(CFM)方法应用于序列建模,利用基于信息几何的插值技术,具体采用对数空间中的Kullback-Leibler(KL)散度测地线,对应于对数空间的线性插值。我们严格证明,在此设定下最大化条件似然可精确恢复流匹配的速度场,为该方法提供了理论依据。为解决基础推理中的性能差距,提出一种迭代去噪-重加噪的新型采样策略,并设计一种融合该策略与基础过程的混合方案。在无条件及条件文本与代码补全任务中,该方法在相同设置下相较先前NAR基线,在困惑度和下游指标上均有提升。
原文摘要 · Abstract (English)
Non-autoregressive (NAR) language models offer notable efficiency in text generation by circumventing the sequential bottleneck of autoregressive decoding. However, accurately modeling dependencies in discrete sequences remains challenging in this paradigm. In this work, we advance the field of NAR generation by applying conditional flow matching (CFM) methods grounded in geometrically principled interpolation, specifically leveraging Kullback-Leibler (KL) divergence geodesics, which correspond to linear interpolation in logit space. We rigorously establish that maximizing conditional likelihood in this setting precisely recovers the flow matching velocity field, supplying the theoretical justification for this approach in sequence modeling. To address practical performance gaps of basic inference, we propose a novel empirical sampling strategy that iteratively denoises and re-noises, along with a hybrid scheme that integrates our sampling method with basic procedure. Across unconditional and conditional text and code infilling, the approach improves perplexity and downstream metrics over prior NAR baselines under matched settings.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。