不训练即可加速统一模型,生成和理解任务分别优化。
Flash-Unified: A Training-Free and Task-Aware Acceleration Framework for Native Unified Models
- 按任务类型动态剪枝层与令牌,消除冗余计算。
- 生成任务提速1.78至2.01倍,理解任务保持顶级性能。
- 适合需要高效部署多模态统一模型的场景。
原生统一多模态模型整合生成与理解能力,但计算开销大,难以实际部署。现有加速方法多采用静态、单一策略,忽略生成(如图像生成)与理解(如VQA)在计算需求上的本质差异。本文首次系统分析统一模型,发现显著参数专业化:不同神经元组对各任务至关重要,表明统一模型在架构内隐式构建了分离的推理路径。基于此,提出无需训练、任务感知的加速框架FlashU,针对不同任务定制优化。对两类任务均引入任务特定网络剪枝与动态层跳过,以消除层间及任务特异性冗余。视觉生成中,通过时变引导尺度控制与扩散头缓存实现时间近似;多模态理解中,在剪枝模型基础上,利用V-Norm代理实现动态令牌剪枝,挖掘视觉输入空间冗余。在Show-o2数据集上,FlashU在生成与理解任务中均实现1.78×至2.01×的推理加速,同时保持最先进性能,优于对比的统一模型,验证了任务感知加速范式。代码已开源。
原文摘要 · Abstract (English)
Native unified multimodal models, which integrate both generative and understanding capabilities, face substantial computational overhead that hinders their real-world deployment. Existing acceleration techniques typically employ a static, monolithic strategy, ignoring the fundamental divergence in computational profiles between iterative generation tasks (e.g., image generation) and single-pass understanding tasks (e.g., VQA). In this work, we present the first systematic analysis of unified models, revealing pronounced parameter specialization, where distinct neuron sets are critical for each task. This implies that, at the parameter level, unified models have implicitly internalized separate inference pathways for generation and understanding within a single architecture. Based on these insights, we introduce a training-free and task-aware acceleration framework, FlashU, that tailors optimization to each task's demands. Across both tasks, we introduce Task-Specific Network Pruning and Dynamic Layer Skipping, aiming to eliminate inter-layer and task-specific redundancy. For visual generation, we implement a time-varying control signal for the guidance scale and a temporal approximation for the diffusion head via Diffusion Head Cache. For multimodal understanding, building upon the pruned model, we introduce Dynamic Token Pruning via a V-Norm Proxy to exploit the spatial redundancy of visual inputs. Extensive experiments on Show-o2 demonstrate that FlashU achieves 1.78$\times$ to 2.01$\times$ inference acceleration across both understanding and generation tasks while maintaining SOTA performance, outperforming competing unified models and validating our task-aware acceleration paradigm. Our code is publicly available at https://github.com/Rirayh/FlashU.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。