通过分层缓存优化流匹配推理,显著提速且保持高质量图像生成。
LayerCache: Exploiting Layer-wise Velocity Heterogeneity for Efficient Flow Matching Inference
- 按层分组动态决定缓存策略,利用不同层速度变化差异。
- 在1024x1024图像上实现1.37倍加速,PSNR提升5.38dB,LPIPS降低70%。
- 适合追求高效高质图像生成的开发者与研究者使用。
流匹配模型虽能生成顶级质量的图像,但因依赖大型Transformer进行多步去噪而带来高昂的推理开销。我们观察到,Transformer中不同层级组的速度动态存在显著差异:浅层高度稳定,适合激进缓存;深层则速度变化剧烈,需完整计算。现有缓存方法将整个Transformer视为单一整体,每步仅做统一缓存决策,未能利用这一异构性。基于此,我们提出LayerCache——一种分层感知的缓存框架,将Transformer划分为多个层级组,并在每步去噪中独立决定各组缓存策略。该方法引入自适应的JVP跨度K选择机制,依据各组稳定性评估平衡估计精度与计算节省。我们构建了一个覆盖时间步、层级组和JVP跨度的三维调度问题,并采用贪心预算分配算法求解。在Qwen-Image(1024x1024,50步)上,LayerCache实现PSNR 37.46 dB(较MeanCache提升5.38 dB)、SSIM 0.9834、LPIPS 0.0178(较MeanCache降低70%),同时获得1.37倍加速,在质量-速度帕累托前沿全面超越此前所有缓存方法。
原文摘要 · Abstract (English)
Flow Matching models achieve state-of-the-art image generation quality but incur substantial inference cost due to iterative denoising through large Transformer networks. We observe that different layer groups within a Transformer exhibit markedly heterogeneous velocity dynamics: shallow layers are highly stable and amenable to aggressive caching, while deep layers undergo large velocity changes that demand full computation. Existing caching methods, however, treat the entire Transformer as a monolithic unit, applying a single caching decision per timestep and thus failing to exploit this heterogeneity. Based on this finding, we propose LayerCache, a layer-aware caching framework that partitions the Transformer into layer groups and makes independent, per-group caching decisions at each denoising step. LayerCache introduces an adaptive JVP span K selection mechanism that leverages per-group stability measurements to balance estimation accuracy and computational savings. We formulate a three-dimensional scheduling problem over timesteps, layer groups, and JVP span, and solve it with a greedy budget allocation algorithm. On Qwen-Image (1024x1024, 50 steps), LayerCache achieves PSNR 37.46 dB (+5.38 dB over MeanCache), SSIM 0.9834, and LPIPS 0.0178 (a 70% reduction over MeanCache) at 1.37x speedup, dominating all prior caching methods on the quality-speed Pareto frontier.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。