arXiv:2606.02133cs.LGcs.AI2026-06被引 1

让生成顺序可学习,提升非单调生成的灵活性与效果

Variational Learning for Insertion-based Generation

论文配图:Variational Learning for Insertion-based Generation
图 1 · 摘自论文原文
  • 基于排列的变分推断,动态学习插入位置与顺序
  • 支持可变长度生成,比固定网格方法更灵活
  • 在分子序列和规划任务中提升建模质量与泛化能力

非单调序列生成方法(如掩码扩散模型)通过允许以非固定顺序生成词元,为左到右自回归建模提供了灵活替代。尽管具有实用优势,现有大多数非单调模型是顺序无关的,依赖固定长度网格,限制了对可变长度生成和自适应插入顺序的支持。本文提出一种概率框架,用于学习可变长度插入模型中的插入顺序。我们建立插入轨迹与排列之间的双射关系,实现数据似然的精确重参数化,表示为所有排列之和。基于此,我们提出插入过程(Insertion Process, IP),一种联合学习插入位置、内容及终止时机的随机生成模型,通过基于排列的变分推断进行训练。与以往固定画布方法不同,IP 原生支持可变长度生成,并学习数据驱动的插入顺序偏好。在目标条件规划和分子字符串生成任务上的实验表明,学习插入顺序能显著提升建模质量和泛化能力,尤其在无标准左到右结构的领域。

原文摘要 · Abstract (English)

Non-monotonic sequence generation methods, such as masked diffusion models, provide a flexible alternative to left-to-right autoregressive modeling by allowing tokens to be generated in non-fixed and prescribed orders. Despite their practical advantages, most existing non-monotonic models are order-agnostic and rely on a fixed-length grid, limiting their ability to support variable-length generation and adaptive insertion order. In this work, we introduce a probabilistic framework for learning insertion order in variable-length insertion models. We formalize a bijective correspondence between insertion trajectories and permutations, which enables an exact reparameterization of the data likelihood as a sum over permutations. Building on this result, we propose the Insertion Process (IP), a stochastic generative model that jointly learns where to insert, what to insert, and when to terminate, trained via permutation-based variational inference. Unlike prior fixed-canvas approaches, IP natively supports variable-length generation and learns data-driven preferences over insertion orders. Experiments on goal-conditioned planning and molecular string generation demonstrate that learning insertion order improves both modeling quality and generalization in domains without a canonical left-to-right structure.

生成模型插入生成变分推断可变长度

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。