通过位置感知剪枝,让超大扩散模型在手机上也能流畅生成高清图。
HierarchicalPrune: Position-Aware Compression for Large-Scale Diffusion Models
- 按功能层级剪掉冗余的后期处理层,保留关键语义结构。
- 压缩后内存减至3.2GB,推理速度提升超38%,质量损失仅2.6%。
- 适合移动端部署,用户实测认为画质与原模型无明显差异。
当前最先进的文生图扩散模型参数量达80亿至110亿,难以在资源受限设备上运行。本文提出HierarchicalPrune,基于一个核心发现:扩散模型各模块具有功能层级——早期模块构建语义结构,后期模块负责纹理细化。该框架融合三项技术:(1) 位置感知剪枝,根据层级关系移除低效后期模块;(2) 位置权重保护,系统性保留对语义完整性至关重要的早期部分;(3) 敏感度引导蒸馏,依据模块敏感度差异动态调整知识迁移强度。结合INT4量化,该方法在服务器与消费级GPU上实现77.5%-80.4%内存压缩(如从15.8GB降至3.2GB),延迟降低27.9%-38.0%,生成评估分数(GenEval)下降最少2.6%,HPSv2得分下降7%。85名用户的综合测评表明,其感知质量接近原模型,显著优于已有方法。
原文摘要 · Abstract (English)
State-of-the-art text-to-image diffusion models (DMs) achieve remarkable quality, yet their massive parameter scale (8-11B) poses significant challenges for inferences on resource-constrained devices. In this paper, we present HierarchicalPrune, a novel compression framework grounded in a key observation: DM blocks exhibit distinct functional hierarchies, where early blocks establish semantic structures while later blocks handle texture refinements. HierarchicalPrune synergistically combines three techniques: (1) Hierarchical Position Pruning, which identifies and removes less essential later blocks based on position hierarchy; (2) Positional Weight Preservation, which systematically protects early model portions that are essential for semantic structural integrity; and (3) Sensitivity-Guided Distillation, which adjusts knowledge-transfer intensity based on our discovery of block-wise sensitivity variations. As a result, our framework brings billion-scale diffusion models into a range more suitable for on-device inference, while preserving the quality of the output images. Specifically, combined with INT4 weight quantisation, HierarchicalPrune achieves 77.5-80.4% memory footprint reduction (e.g., from 15.8 GB to 3.2 GB) and 27.9-38.0% latency reduction, measured on server and consumer grade GPUs, with the minimum drop of 2.6% in GenEval score and 7% in HPSv2 score compared to the original model. Finally, our comprehensive user study with 85 participants demonstrates that HierarchicalPrune maintains perceptual quality comparable to the original model while significantly outperforming prior works.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。