让图像生成模型自动分配计算资源,更省算力还更清晰。
DC-DiT: Adaptive Compute and Elastic Inference for Visual Generation via Dynamic Chunking
- 用动态分块机制替代固定分块,按需压缩图像信息。
- 推理时减少36.8%算力,FID提升37.8%,画质更优。
- 支持灵活计算预算调整,适合部署在资源受限场景。
扩散Transformer依赖静态的图像分块方式,对平滑背景、细节区域、噪声早期步和后期优化阶段均使用相同令牌预算。本文提出动态分块扩散Transformer(DC-DiT),以端到端学习的编码器-路由器-解码器结构替代固定分块,通过可学习的分块机制自适应压缩二维输入为更短的令牌序列。该方法在可预测区域和噪声早期步骤中减少令牌数量,在细节区域和后期优化阶段增加令牌数量,实现无需监督的空间分割与时间自适应压缩策略。此外,路由器提供保留令牌的重要性排序,支持弹性推理:单个检查点可在不同计算预算下灵活评估,实现质量与计算量的平滑权衡。DC-DiT可由预训练DiT模型迁移而来,且兼容其他动态计算方法。在类别条件图像生成任务中,相比基线DiT模型,最多减少36.8%推理浮点运算量,同时将FID提升37.8%,在多种模型规模、分辨率与引导设置下均获得更强的质量-算力帕累托前沿。结果表明,自适应令牌化是提升视觉生成效率与推理灵活性的通用机制。
原文摘要 · Abstract (English)
Diffusion Transformers rely on static patchify tokenization, assigning the same token budget to smooth backgrounds, detailed object regions, noisy early timesteps, and late-stage refinements. We introduce the Dynamic Chunking Diffusion Transformer (DC-DiT), which replaces fixed patchification with a learned encoder-router-decoder scaffold that adaptively compresses the 2D input into a shorter token sequence through a chunking mechanism learned end-to-end with diffusion training. DC-DiT allocates fewer tokens to predictable regions and noisy timesteps, and more tokens to detailed regions and later refinement stages, yielding meaningful spatial segmentations and timestep-adaptive compression schedules without supervision. Furthermore, the router provides an importance ordering over retained tokens, enabling elastic inference: a single checkpoint can be evaluated at flexible compute budgets with a smooth quality-compute tradeoff. Additionally, DC-DiT can be upcycled from pretrained DiT checkpoints and is also compatible with orthogonal dynamic computation approaches. On class-conditional ImageNet generation, DC-DiT reduces inference FLOPs by up to 36.8% and improves FID by up to 37.8% over DiT baselines, yielding a stronger quality--compute Pareto frontier across model scales, resolutions, and guidance settings. More broadly, these results suggest that adaptive tokenization is a general mechanism for making visual generation both more efficient and more flexible at inference time.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。