动态分块提升图像生成效率，减少计算量并加快收敛。
DPAR: Dynamic Patchification for Efficient Autoregressive Visual Generation
- 根据信息熵动态合并图像片段为可变大小块，替代固定分块。
- 在ImageNet 256/384上分别减少1.81倍和2.06倍令牌数，训练耗能降40%。
- 适合追求高效视觉生成与多模态融合的开发者使用。
解码器仅架构的自回归图像生成通常依赖固定长度的标记化方案，其标记数量随分辨率呈平方增长，显著增加注意力计算与内存开销。本文提出DPAR，一种新型解码器仅自回归模型，通过动态聚合图像标记为可变数量的块，实现高效图像生成。我们的工作首次证明，轻量级无监督自回归模型的下一标记预测熵可作为基于信息内容合并标记为更大块的可靠准则。DPAR对标准解码器架构改动极小，确保与多模态生成框架兼容，并将更多算力分配给高信息区域生成。此外，我们证明以动态尺寸块训练可获得对块边界鲁棒的表示，使DPAR可在推理时扩展至更大块尺寸。DPAR在ImageNet 256和384生成分辨率下分别降低1.81倍和2.06倍标记数，训练成本最高降低40% FLOPs。同时，该方法收敛更快，且相比基线模型FID改善达27.1%。
原文摘要 · Abstract (English)
Decoder-only autoregressive image generation typically relies on fixed-length tokenization schemes whose token counts grow quadratically with resolution, substantially increasing the computational and memory demands of attention. We present DPAR, a novel decoder-only autoregressive model that dynamically aggregates image tokens into a variable number of patches for efficient image generation. Our work is the first to demonstrate that next-token prediction entropy from a lightweight and unsupervised autoregressive model provides a reliable criterion for merging tokens into larger patches based on information content. DPAR makes minimal modifications to the standard decoder architecture, ensuring compatibility with multimodal generation frameworks and allocating more compute to generation of high-information image regions. Further, we demonstrate that training with dynamically sized patches yields representations that are robust to patch boundaries, allowing DPAR to scale to larger patch sizes at inference. DPAR reduces token count by 1.81x and 2.06x on Imagenet 256 and 384 generation resolution respectively, leading to a reduction of up to 40% FLOPs in training costs. Further, our method exhibits faster convergence and improves FID by up to 27.1% relative to baseline models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。