通过递归复用模型参数,提升掩码扩散模型的生成效率。
Recursive Scaling in Masked Diffusion Models

- 在每步去噪中重复使用同一Transformer,实现参数复用的递归深度扩展。
- 递归迭代次数为L时,性能相当于非递归模型参数量增加约L倍。
- 减少推理时前向传播次数,兼顾参数效率与计算资源分配。
掩码扩散模型(MDMs)是序列生成的新兴范式。传统缩放方法通过增加参数量或去噪步数实现。本文提出递归掩码扩散模型(R-MDMs),引入递归深度作为第三种缩放维度:在每一步去噪中重复应用相同的去噪Transformer。递归机制通过参数复用实现输出的迭代优化,提升有效模型深度而不增加参数量。在数独、倒计时等结构化生成任务中,具有L次递归迭代的模型通常达到非递归基线在约L×参数量下的性能。此外,递归细化可部分替代额外去噪步数,使递归模型以更少前向传播次数达到相同生成质量。结果表明,递归深度是MDMs中实用的缩放机制,同时提升参数效率与测试时间算力分配。
原文摘要 · Abstract (English)
Masked diffusion models (MDMs) have recently emerged as a promising paradigm for sequence generation. Scaling MDMs is conventionally achieved by increasing the parameter count or the number of denoising steps. We introduce Recursive Masked Diffusion Models (R-MDMs), which add recursive depth as a third scaling axis by repeatedly applying the same denoising transformer within each diffusion step. Recursion enables iterative refinement of the output through parameter reuse, increasing effective model depth without increasing parameter count. Across structured generation tasks, including Sudoku and Countdown, we show that R-MDMs achieve substantially improved parameter efficiency: a model with $L$ recursive iterations often matches the performance of non-recursive baselines with roughly $L\times$ more parameters. Moreover, recursive refinement can partially substitute for additional denoising steps, allowing recursive models to reach the same generation quality with fewer forward passes at inference time. These results suggest that recursive depth is a practically useful scaling mechanism for MDMs, improving both parameter efficiency and the allocation of test-time compute.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。