提出GTL方法,让离散扩散模型在小数据下也能高效迁移,不改原模型。
Guided Transfer Learning for Discrete Diffusion Models
- 设计调度机制,使离散扩散模型迁移计算量线性增长,避免词汇表爆炸。
- 小数据时性能优于全参数微调,大样本时后者仍更优。
- 适合数据稀缺场景,如低资源语言建模或短序列生成任务。
离散扩散模型(DMs)在语言等离散领域表现优异,但通常依赖大规模训练数据,在小数据场景下性能受限。现有连续扩散模型的迁移学习方法通过分类器比率引导实现有效迁移,但离散情形尚未探索。本文分析发现,直接扩展该方法在计算上不可行,复杂度随词表规模呈指数增长。为此提出调度机制,构建适用于离散扩散模型的引导迁移学习(GTL)算法。GTL无需修改预训练去噪器,将计算成本降至线性,支持更长序列生成。在合成马尔可夫链和语言建模任务上评估,结果表明:目标数据量小时GTL显著优于全参数微调;数据量大时后者更优。此外实验揭示关键缺陷:当源与目标分布重叠差时,分类器比率不可靠,导致迁移失效。
原文摘要 · Abstract (English)
Discrete diffusion models (DMs) have achieved strong performance in language and other discrete domains, offering a compelling alternative to autoregressive modeling. Yet this performance typically depends on large training datasets, challenging the performance of DMs in small-data regimes -- common under real-world constraints. Aimed at this challenge, recent work in continuous DMs suggests that transfer learning via classifier ratio-based guidance can adapt a pretrained DM to a related target distribution, often outperforming alternatives such as full-weight fine-tuning on the target data. By contrast, transfer learning for discrete DMs remains unexplored. We address this gap by exploring practical analogues of ratio-based transfer learning for discrete DMs. Our theoretical analysis shows that a direct extension of existing ratio-based guidance is computationally prohibitive, scaling with vocabulary size. To overcome this limitation, we introduce a scheduling mechanism that yields a practical algorithm, Guided Transfer Learning for discrete diffusion models (GTL). GTL enables sampling from a target distribution without modifying the pretrained denoiser and reduces the cost to linear scaling in vocabulary size, which in turn supports longer sequence generation. We evaluate GTL on sequential data, including synthetic Markov chains and language modeling tasks, and provide a detailed empirical analysis of its behavior. The results highlight a clear trade-off: when target datasets are large, weight fine-tuning is often preferable, whereas GTL becomes increasingly effective as target data shrinks. Finally, we experimentally demonstrate a key failure mode of GTL: when the source and target distributions overlap poorly, the ratio-based classifier required for guidance becomes unreliable, limiting transfer performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。