提出可自适应调度噪声的文本生成模型,让生成更贴合语境。
Meta-DiffuB: A Contextualized Sequence-to-Sequence Text Diffusion Model with Meta-Exploration
- 用元探索训练动态噪声调度器,实现上下文感知的噪声控制。
- 在4个基准数据集上超越现有扩散模型和预训练语言模型性能。
- 调度器可直接套用,无需微调,适合快速集成到现有生成系统。
扩散模型作为一种新兴生成范式,在图像、音频、视频和文本生成中取得显著进展。通过DiffuSeq,该方法已被用于序列到序列文本生成(Seq2Seq),称为S2S扩散模型。现有S2S扩散模型主要依赖固定或人工设计的噪声调度规则,但这些方法存在非上下文化的噪声问题,未能充分考虑序列生成任务的特性。本文提出Meta-DiffuB框架——一种基于元探索的新型调度器-利用器协同架构,旨在克服上述局限。我们使用元探索训练一个额外的调度器模型,为每个句子动态生成上下文相关的噪声调度策略。利用器模型作为S2S扩散模型,依据调度器提供的噪声进行更新与生成。Meta-DiffuB在四个主流的Seq2Seq基准数据集上均达到当前最优性能,优于先前的S2S扩散模型及微调的预训练语言模型(PLMs)。我们进一步分析并可视化了噪声调度对不同难度句子生成的影响。此外,调度器模型可作为“即插即用”模块,无需推理阶段微调即可增强DiffuSeq,提升部署灵活性。
原文摘要 · Abstract (English)
The diffusion model, a new generative modeling paradigm, has achieved significant success in generating images, audio, video, and text. It has been adapted for sequence-to-sequence text generation (Seq2Seq) through DiffuSeq, termed S2S Diffusion. Existing S2S-Diffusion models predominantly rely on fixed or hand-crafted rules to schedule noise during the diffusion and denoising processes. However, these models are limited by non-contextualized noise, which fails to fully consider the characteristics of Seq2Seq tasks. In this paper, we propose the Meta-DiffuB framework - a novel scheduler-exploiter S2S-Diffusion paradigm designed to overcome the limitations of existing S2S-Diffusion models. We employ Meta-Exploration to train an additional scheduler model dedicated to scheduling contextualized noise for each sentence. Our exploiter model, an S2S-Diffusion model, leverages the noise scheduled by our scheduler model for updating and generation. Meta-DiffuB achieves state-of-the-art performance compared to previous S2S-Diffusion models and fine-tuned pre-trained language models (PLMs) across four Seq2Seq benchmark datasets. We further investigate and visualize the impact of Meta-DiffuB's noise scheduling on the generation of sentences with varying difficulties. Additionally, our scheduler model can function as a "plug-and-play" model to enhance DiffuSeq without the need for fine-tuning during the inference stage.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。