提出新型离散流匹配优化方法，显著减少状态转移次数。
Minibatch Optimal Transport and Perplexity Bound Estimation in Discrete Flow Matching
- 基于最小化运输成本的动态最优传输策略，支持小批量优化
- 状态转移次数从1024降至32，生成困惑度不变且多样性不降
- 设计困惑度上界估计，支持模型训练与评估，适合语言建模等场景
离散流匹配是一种新兴的分类数据建模框架，表现可媲美自回归模型。然而，由于离散路径的随机性，无法使用连续流匹配中的校正策略，需采用替代方法减少状态转移。本文提出一种类动态最优传输的最小化目标，并推导出具有凸插值的离散流的Kantorovich形式，其中运输成本仅依赖于状态间差异，可通过小批量策略优化。结果表明，该方法可将状态转移次数最多减少至原来的1/32（从1024降至32），在保持生成困惑度的同时不牺牲多样性。此外，离散流路径的非确定性导致无法建立瞬时变量变换对应关系，难以获得如连续流般精确的概率估计。为此，本文提出两种困惑度上界，实现有原则的训练、评估与模型比较。最后，引入Multimask Flows，在生成困惑度上优于掩码流，尤其在结合小批量最优传输时表现更优。
原文摘要 · Abstract (English)
Discrete flow matching, a recent framework for modeling categorical data, has shown competitive performance with autoregressive models. However, unlike continuous flow matching, the rectification strategy cannot be applied due to the stochasticity of discrete paths, necessitating alternative methods to minimize state transitions. We propose a dynamic-optimal-transport-like minimization objective and derive its Kantorovich formulation for discrete flows with convex interpolants, where transport cost depends solely on inter-state dissimilarity and can be optimized via minibatch strategies. We show that such methods can reduce the number of transitions up to 32 times (1024 to 32) to reach the same generative perplexity without compromising diversity. Additionally, path nondeterminism in discrete flows precludes an instantaneous change-of-variables analogue, preventing precise probability estimation available to continuous flows. We therefore propose two upper bounds on perplexity, enabling principled training, evaluation and model comparison. Finally, we introduce Multimask Flows which outperform masked flows in generative perplexity without compromising diversity, particularly when utilizing minibatch Optimal Transport.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。