arXiv:2505.14709cs.CVcs.AI2025-05被引 13

利用帧间冗余加速边缘端自回归视频生成,速度提升超2.1倍。

FastCar: Cache Attentive Replay for Fast Auto-Regressive Video Generation on the Edge

  • 通过时序注意力评分判断是否复用前帧MLP缓存结果
  • 在边缘设备上实现2.1倍以上解码加速与更高能效
  • 适合高分辨率长时视频生成,尤其适配资源受限场景

自回归(AR)模型在语言生成中表现优异,近年来在视觉生成任务中也展现出潜力,因其采样效率高。然而,视频生成需大量标记以保证时间连贯性,导致解码阶段开销显著。我们观察到:(i) 解码阶段的MLP模块主导推理延迟;(ii) 相邻帧的MLP输出存在高度时序冗余。为此,提出FastCar框架,通过挖掘时序冗余加速解码。设计时序注意力评分(TAS),理论分析并决定是否采用重放策略(即复用前帧缓存的MLP输出以减少重复计算)。同时,在FPGA上开发基于TAS的动态资源调度(DRS)硬件加速器,提升资源利用率与推理速度。实验表明,本方法相较传统稀疏注意力提升2.1倍以上解码速度,且能效更高。结合稀疏注意力后,可缓解漂移问题,显著提升高分辨率、长时视频生成性能。代码已开源。

原文摘要 · Abstract (English)

Auto-regressive (AR) models, initially successful in language generation, have recently shown promise in visual generation tasks due to their superior sampling efficiency. Unlike image generation, video generation requires a substantially larger number of tokens to produce coherent temporal frames, resulting in significant overhead during the decoding phase. Our key observations are: (i) MLP modules in the decode phase dominate the inference latency, and (ii) there exists high temporal redundancy in MLP outputs of adjacent frames. In this paper, we propose the \textbf{FastCar} framework to accelerate the decode phase for the AR video generation by exploring the temporal redundancy. The Temporal Attention Score (TAS) is proposed to determine whether to apply the replay strategy (\textit{i.e.}, reusing cached MLP outputs from the previous frame to reduce redundant computations) with detailed theoretical analysis and justification. Also, we develop a hardware accelerator on FPGA with Dynamic Resource Scheduling (DRS) based on TAS to enable better resource utilization and faster inference. Experimental results demonstrate the effectiveness of our method, which outperforms traditional sparse attention approaches with more than 2.1x decoding speedup and higher energy efficiency on the edge. Furthermore, by combining FastCar and sparse attention, FastCar can boost the performance of sparse attention with alleviated drifting, demonstrating our unique advantages for high-resolution and long-duration video generation. Code: https://github.com/shawnricecake/fast-car

视频生成自回归边缘计算加速

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。