通过动态裁剪冗余掩码令牌，实现多模态扩散模型推理速度翻倍。
Sparse-LaViDa: Sparse Multimodal Discrete Diffusion Language Models
- 动态裁剪无用掩码令牌，减少重复计算
- 引入专用寄存器令牌保持生成质量，速度提升最高2倍
- 适配训练与推理一致性，适合图像生成与编辑任务
掩码离散扩散模型（MDMs）在图像理解、生成和编辑等多模态任务中表现优异，但其推理速度受限于每步采样需重复处理冗余掩码令牌。本文提出Sparse-LaViDa，一种新型建模框架，在每次推理步骤中动态裁剪不必要的掩码令牌以加速采样。为保持生成质量，引入专用注册令牌作为被裁剪令牌的紧凑表征。同时设计专用注意力掩码，确保训练阶段与裁剪采样过程一致。基于当前最先进的统一多模态扩散模型LaViDa-O，Sparse-LaViDa在文本到图像生成、图像编辑和数学推理等多种任务中实现最高2倍的速度提升，且生成质量不受影响。
原文摘要 · Abstract (English)
Masked Discrete Diffusion Models (MDMs) have achieved strong performance across a wide range of multimodal tasks, including image understanding, generation, and editing. However, their inference speed remains suboptimal due to the need to repeatedly process redundant masked tokens at every sampling step. In this work, we propose Sparse-LaViDa, a novel modeling framework that dynamically truncates unnecessary masked tokens at each inference step to accelerate MDM sampling. To preserve generation quality, we introduce specialized register tokens that serve as compact representations for the truncated tokens. Furthermore, to ensure consistency between training and inference, we design a specialized attention mask that faithfully matches the truncated sampling procedure during training. Built upon the state-of-the-art unified MDM LaViDa-O, Sparse-LaViDa achieves up to a 2x speedup across diverse tasks including text-to-image generation, image editing, and mathematical reasoning, while maintaining generation quality.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。