arXiv:2411.06119cs.CVcs.LG2024-11

提出可复用固定结构的扩散模型,适配手机等设备部署。

Hardware-Friendly Diffusion Models with Fixed-Size Reusable Structures for On-Device Image Generation

  • 用固定大小可复用的Transformer块替代传统架构
  • 在CelebA上实现1.6的FID,性能达顶尖水平
  • 无位置编码、无令牌设计,适合移动设备

视觉Transformer和U-Net架构广泛应用于扩散模型实现,但均存在设备端部署难题。视觉Transformer依赖位置编码维持标记对应关系,虽具固定大小可复用块优势;而U-Net缺乏此特性,其降噪主干使用变尺寸的下采样与上采样块。为此,本文提出一种以固定大小可复用Transformer块为核心的新架构,具备低复杂度、无令牌设计、无位置编码、一致性和可扩展性,更利于硬件实现。该模型在无条件与有条件图像生成任务中表现优异且稳定,在无条件生成任务中于CelebA数据集上取得1.6的先进FID得分。

原文摘要 · Abstract (English)

Vision Transformers and U-Net architectures have been widely adopted in the implementation of Diffusion Models. However, each architecture presents specific challenges while realizing them on-device. Vision Transformers require positional embedding to maintain correspondence between the tokens processed by the transformer, although they offer the advantage of using fixed-size, reusable repetitive blocks following tokenization. The U-Net architecture lacks these attributes, as it utilizes variable-sized intermediate blocks for down-convolution and up-convolution in the noise estimation backbone for the diffusion process. To address these issues, we propose an architecture that utilizes a fixed-size, reusable transformer block as a core structure, making it more suitable for hardware implementation. Our architecture is characterized by low complexity, token-free design, absence of positional embeddings, uniformity, and scalability, making it highly suitable for deployment on mobile and resource-constrained devices. The proposed model exhibit competitive and consistent performance across both unconditional and conditional image generation tasks. The model achieved a state-of-the-art FID score of 1.6 on unconditional image generation with the CelebA.

扩散模型轻量化移动端

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。