让扩散模型在手机上快速生成图像,不依赖云端。
EdgeDiT: Hardware-Aware Diffusion Transformers for Efficient On-Device Image Generation
- 针对手机NPU设计轻量版扩散模型,剪掉冗余结构
- 参数减少20-30%,计算量降36-46%,延迟降1.65倍
- 适合需要隐私、离线运行的移动端AI生成应用
扩散变压器(DiT)在高保真图像生成中达到新水平,但其巨大的计算复杂度和内存需求阻碍了在资源受限的边缘设备上的本地部署。本文提出EdgeDiT,一组专为移动神经处理单元(NPUs)如高通Hexagon和苹果神经引擎(ANE)设计的硬件高效生成变压器。通过硬件感知优化框架,系统性识别并修剪对移动数据流特别负担的DiT主干结构冗余。该方法生成一系列轻量级模型,在不牺牲原始变换器架构的扩展优势和表达能力的前提下,实现参数减少20-30%、浮点运算量(FLOPs)降低36-46%、设备端延迟降低1.65倍。大量基准测试表明,EdgeDiT在生成质量(以弗雷谢入学距离,FID衡量)与推理延迟之间提供了优于优化后的移动U-Nets及原始DiT变体的帕累托最优权衡。通过实现响应迅速、私密且离线的生成式AI,EdgeDiT为将大规模基础模型从高端GPU迁移至用户掌上设备提供了可扩展蓝图。
原文摘要 · Abstract (English)
Diffusion Transformers (DiT) have established a new state-of-the-art in high-fidelity image synthesis; however, their massive computational complexity and memory requirements hinder local deployment on resource-constrained edge devices. In this paper, we introduce EdgeDiT, a family of hardware-efficient generative transformers specifically engineered for mobile Neural Processing Units (NPUs), such as the Qualcomm Hexagon and Apple Neural Engine (ANE). By leveraging a hardware-aware optimization framework, we systematically identify and prune structural redundancies within the DiT backbone that are particularly taxing for mobile data-flows. Our approach yields a series of lightweight models that achieve a 20-30% reduction in parameters, a 36-46% decrease in FLOPs, and a 1.65-fold reduction in on-device latency without sacrificing the scaling advantages or the expressive capacity of the original transformer architecture. Extensive benchmarking demonstrates that EdgeDiT offers a superior Pareto-optimal trade-off between Frechet Inception Distance (FID) and inference latency compared to both optimized mobile U-Nets and vanilla DiT variants. By enabling responsive, private, and offline generative AI directly on-device, EdgeDiT provides a scalable blueprint for transitioning large-scale foundation models from high-end GPUs to the palm of the user.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。