连续潜空间生成需优化解码可恢复性,仅靠几何匹配不够。
When Latent Geometry Is Not Enough: Draft-Conditioned Latent Refinement for Non-Autoregressive Text Generation

- 用草稿条件的局部潜变量精炼替代从噪声生成
- 768维潜变量解码准确率高达0.938,256维显著下降
- 适合关注非自回归文本生成解码质量的研究者
连续扩散与流模型在非自回归文本生成中因可并行更新所有位置而具有吸引力。主要挑战在于连续潜状态与离散词元之间的接口。本研究构建了一个基于冻结BERT编码器、并行解码器、去噪草稿先验(DraftPrior)、局部FlowNet和学习的对角度量网络(MetricNet)的草稿条件潜变量精炼模型。早期高斯起点实验表明,良好的潜空间度量(如尺度匹配或余弦相似度)并不能保证良好解码。生成的潜变量虽接近真实编码潜变量,仍会产生高熵、有偏或重复的词元分布。因此,将任务重新定义为受控的局部精炼而非从噪声完全生成。在ROCStories数据集上,以前两句为提示、后三句为目标,使用完整的768维BERT潜变量比压缩的256维潜变量恢复词元效果更好。768维潜变量下,草稿先验的目标词元概率分别为:干净草稿0.938,3%词元丢失时0.613,5%时0.483,10%时0.272。局部流精炼与融合解码器感知读出带来小幅提升,而度量学习与OT式对齐改善几何结构但无法弥合解码差距。核心结论是:潜空间几何本身不足以支撑生成质量。连续潜变量文本生成应通过解码可恢复性、初始分布质量以及精炼是否保持解码可读结构来评估。
原文摘要 · Abstract (English)
Continuous diffusion and flow models are attractive for non-autoregressive text generation because they can update all positions in parallel. A major difficulty is the interface between continuous latent states and discrete tokens. This report studies a draft-conditioned latent refinement model built from a frozen BERT encoder, a parallel decoder, a denoising DraftPrior, a local FlowNet, and a learned diagonal MetricNet. Early Gaussian-start experiments showed that good latent-space metrics, such as scale matching or cosine similarity, do not guarantee good decoding. Generated latents can be close to real encoder latents but still produce high-entropy, biased, or repetitive token distributions. We therefore frame the task as controlled local refinement rather than full generation from noise. On ROCStories, using the first two sentences as prompt and the last three as target, full 768-dimensional BERT latents recover tokens much better than compressed 256-dimensional latents. With 768-dimensional latents, DraftPrior target-token probability is 0.938 for clean drafts, 0.613 for 3% token dropout, 0.483 for 5% dropout, and 0.272 for 10% dropout. Local flow refinement and fused decoder-aware readout give modest additional gains, while metric learning and OT-style alignment improve geometry but do not close the decoder gap. The main result is a diagnostic one: latent geometry alone is not enough. Continuous latent text generation should be evaluated by decoder recoverability, the quality of the start distribution, and whether refinement preserves decoder-readable structure.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。