提出固定点流模型，提升文本生成质量与效率。
Self-conditioned Flow Map Language Models via Fixed-point Flows
- 从固定点迭代视角重构自条件流模型，统一生成过程
- 在OpenWebText上实现单步与少步生成最优性能
- 适合追求高效高质量文本生成的研究者与开发者
自条件是增强连续流式语言模型的核心技术，通过自身去噪估计来改善生成文本。尽管实证有效，其性能提升机制仍不清晰。此外，基于流映射的少步生成模型如何应用自条件尚不明确。本文揭示自条件流模型本质上求解固定点迭代，从而优化去噪器性能。基于此提出固定点流（fixed-point flows），一个二维自条件流类别，其中一维表示流过程，另一维表示固定点迭代。证明该模型可定义合法流映射，并可通过固定点蒸馏与流映射蒸馏，从自条件流模型中压缩得到。所提出的流动映射语言模型FMLM$^\star$在OpenWebText数据集上，于单步与少步生成任务中超越现有自条件模型与少步模型。代码已公开于https://github.com/Ugness/self-conditioned-fmlm。
原文摘要 · Abstract (English)
Self-conditioning is a core technique that enhances continuous flow-based language models, where the model learns to denoise generated text by conditioning on its own denoising estimate. While empirically successful, its performance improvements are poorly understood. Moreover, there is growing interest in the use of few-step generators based on flow maps, for which how to leverage self-conditioning is unclear. Here, we show that flow language models with self-conditioning solve a fixed-point iteration that bootstraps the performance of the learned denoiser. We use this viewpoint to formulate fixed-point flows, a two-dimensional class of self-conditioned flows, where the first dimension represents the flow process and the second represents the fixed-point iteration. We show that fixed-point flows define valid flow maps, and show that they can be distilled from self-conditioned flow models by compressing both fixed-point iterations and the flow process, the former with fixed-point distillation and the latter with flow map distillation. Our resulting flow map language model, FMLM$^\star$, outperforms state-of-the-art self-conditioned models and few-step models in one- and few-step generation on OpenWebText. Code is available at https://github.com/Ugness/self-conditioned-fmlm.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。