用固定点求解器替代部分去噪器,实现更高效低预算的文本图像生成。
Fixed-Point Masked Generative Modeling

- 用共享注意力层的固定点求解器替代部分去噪器,自适应调整计算深度。
- 在OpenWebText上仅用96次前向传播,困惑度从830.8降至101.8,参数减少38.8%。
- 适合追求低预算高效生成的开发者,支持预训练模型快速转换。
掩码生成模型(MGMs)支持并行解码且跨模态表现优异,但每步需完整双向变换器,训练成本高且低采样预算下质量下降。现有方法通过优化采样器或使用轻量去噪器提升效率,但仍固定分配每步计算量。本文提出固定点掩码生成模型(FP-MGMs),用共享注意力层的固定点求解器替代部分去噪器,实现计算深度自适应。引入跨步一致性损失对齐相邻步骤隐藏表示,并设计三状态复用(3SR)机制,分别处理未改变、仍掩码、新揭示的令牌,以热启动求解器。二者构成完整的训练-推理框架CoFRe。还证明预训练MGM可经短时微调转为FP-MGM,无需重训。在多种模态上,CoFRe显著改善质量与成本权衡:在OpenWebText上,参数减少38.8%,训练时间降11.5%,显存降16.9%,96次前向传播下困惑度从830.8降至101.8;在ImageNette上,训练时间减少48.6%,显存降50.7%,所有采样预算下FID均提升。总体提供了一种更经济的训练与低预算生成方案。
原文摘要 · Abstract (English)
Masked Generative Models (MGMs) enable parallel decoding and achieve strong performance across modalities, but require full-sequence bidirectional transformers at every step, making training costly and degrading quality under low sampling budgets. Existing work improves efficiency via better samplers or cheaper fixed-depth denoisers, but they still allocate a fixed amount of denoiser computation to each refinement step. We introduce Fixed-Point Masked Generative Models (FP-MGMs), which replace part of the denoiser with a fixed-point solver over shared attention layers to enable adaptive depth with fewer parameters. To make it more effective for masked generation, we first introduce a cross-step consistency loss, which aligns hidden representations at neighboring denoising steps and, second, three-state reuse (3SR) which warm-starts the solver using the previous solution by treating differently unchanged, still-masked, and newly revealed tokens respectively. Together, these components define our complete training-to-inference framework for fixed-point masked generation, \emph{CoFRe}. We also show that pre-trained MGMs can be converted into FP-MGMs with short fine-tuning, avoiding full retraining. Across modalities, CoFRe improves the quality and cost trade-off. On OpenWebText, CoFRe reduces parameters by 38.8\%, training time by 11.5\%, and VRAM by 16.9\%, while improving generative perplexity from 830.8 to 101.8 at a budget of $96$ transformer-block forward passes, compared to MDLM. In ImageNette, CoFRe reduces training time by 48.6\% and VRAM by 50.7\%, while improving FID in all sample budgets tested. Overall, CoFRe offers a practical framework for cheaper training and stronger low-budget masked generation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。