通过引入局部上下文信息，提升离散生成模型的采样质量和训练效率。
Context-weighted Discrete Flow Matching
- 基于局部上下文调整连续时间马尔可夫链采样策略。
- 在OpenWebText上将生成困惑度降低63%，媲美强基线模型。
- 无需额外计算开销，支持任意顺序生成，适合高效文本生成场景。
离散流匹配为离散结构的生成建模提供了灵活框架，但标准分解训练目标使模型暴露于难度不一的预测任务中，混合了低熵可预测标记与高熵模糊标记。我们实证发现，每个标记的不确定性与其邻域上下文密度密切相关。受此启发，我们提出对底层连续时间马尔可夫链（CTMC）进行简单修改，引入局部上下文信息。所提出的上下文加权采样器在几乎无额外计算开销下提升了生成质量；同时，缩放交叉熵损失函数对不同标记的训练信号进行重加权，在OpenWebText上将生成困惑度降低高达63%。此外，该方法在生成质量上达到强半自回归块扩散基线水平，同时保持任意顺序生成能力。结果表明，局部上下文是离散生成建模中的关键因素，且简单的上下文感知改进能显著提升采样与训练效率。
原文摘要 · Abstract (English)
Discrete flow matching provides a flexible framework for generative modeling on discrete structures. However, the standard factorized training objective exposes the model to targets of varying difficulty, mixing well-conditioned, predictable tokens with ambiguous, high-entropy ones. We empirically demonstrate that the uncertainty over the value of each token is closely related to the density of available context in its neighborhood. Motivated by this observation, we propose a simple modification to the underlying continuous-time Markov chain (CTMC) that incorporates local context information. Our context-weighted sampler improves generation quality with negligible computational overhead, while our scaled cross-entropy loss function reweights the training signal from different tokens and reduces generative perplexity by up to 63% on OpenWebText. Moreover, our approach matches a strong semi-autoregressive block diffusion baseline in quality while retaining the ability to perform generation in any order. These results highlight the role of local context as an important factor in discrete generative modeling and show that simple context-aware modifications can significantly improve both sampling and training efficiency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。