通过区域令牌接口,让扩散模型用更少的令牌高效生成图像。
Elastic Token Compression for Pixel-Space Diffusion Transformers
- 按图像内容形状切割令牌,保留空间连通性
- 在相同预算下优于现有压缩方法,速度提升2.6倍
- 适合需要高效推理的图像生成应用
自然图像的细节集中在画面的一小部分,但扩散模型在每一层、每一时刻都为每个图像块分配完整令牌,造成巨大浪费,尤其在像素空间模型中更为严重。我们分析一个预训练的像素文本到图像变换器,发现其中间层在图像平坦区域的令牌冗余且呈现连续的内容形状。利用希尔伯特排序可获得具有相同几何结构的令牌序列。连续位置始终是图像邻域,任意连续段构成与内容一致的连通区域,二维分组转化为一维切割。现有压缩方法均会丢失部分信息:相似性合并打散分组,潜在瓶颈丢弃位置,跳过删除应总结的内容。我们在模型特征变化最大的位置进行切割,并将每段聚合为一个区域令牌。我们的区域令牌接口( extit{RTI})使扩散模型适配这些令牌,区域数量在微调时随机抽取,一个检查点可满足不同预算需求。在匹配预算时优于已有方法,在2.0倍速下保持密集生成质量,2.6倍速下仍接近原始表现。代码与模型已开源。
原文摘要 · Abstract (English)
Natural images concentrate their detail in a small fraction of the frame, yet diffusion models spend a full token on every patch, in every layer and at every timestep. The waste is largest in pixel-space models, with no autoencoder to absorb low-level redundancy first. Probing a pretrained pixel text-to-image transformer, we find its middle-block tokens redundant wherever the image is flat. The redundancy occupies connected, content-shaped regions, and exploiting it requires tokens with the same geometry. Cutting a Hilbert ordering of the patches provides them. Consecutive positions are always image neighbours, so any contiguous run is a connected region whose size and shape follow the content, and grouping in two dimensions becomes a cut in one. Existing reductions each lose part of this. Similarity merging scatters its groups, latent bottlenecks discard position, and skipping deletes what it should summarize. We cut where the model's features change most and pool each run into one region token. Our Region Token Interface (\method{}) adapts a diffusion model to these tokens, with the region count drawn at random during fine-tuning so one checkpoint serves every budget. \method{} leads prior reduction methods at matched budgets, matches dense quality at $2.0\times$ the speed, and stays close at $2.6\times$. The code and models are open-sourced at https://eduardzamfir.github.io/rti
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。