动态复用中间层特征，提速文本生成视频且不降质
Foresight: Adaptive Layer Reuse for Accelerated and High-Quality Text-to-Video Generation
- 根据生成过程动态决定哪些层的输出可复用
- 在OpenSora等模型上实现最高4.2倍端到端加速
- 适合需要高效生成高质量视频的研究与应用
扩散Transformer（DiT）在文本到图像、文本到视频生成与编辑任务中表现优异。但其庞大的模型规模和多步去噪过程中时空注意力的二次方计算开销，导致视频生成计算成本高昂。静态缓存虽能通过固定步骤复用特征缓解问题，却无法适应生成动态，造成速度与质量权衡不佳。本文提出Foresight，一种自适应层复用技术，在保持基线性能的前提下，减少去噪步骤间的计算冗余。Foresight动态识别并复用所有层在不同步骤中的块输出，根据分辨率与去噪调度等参数自适应优化效率。应用于OpenSora、Latte和CogVideoX模型，Foresight实现高达\latencyimprv的端到端加速，同时维持视频质量。代码已开源：https://github.com/STAR-Laboratory/foresight。
原文摘要 · Abstract (English)
Diffusion Transformers (DiTs) achieve state-of-the-art results in text-to-image, text-to-video generation, and editing. However, their large model size and the quadratic cost of spatial-temporal attention over multiple denoising steps make video generation computationally expensive. Static caching mitigates this by reusing features across fixed steps but fails to adapt to generation dynamics, leading to suboptimal trade-offs between speed and quality. We propose Foresight, an adaptive layer-reuse technique that reduces computational redundancy across denoising steps while preserving baseline performance. Foresight dynamically identifies and reuses DiT block outputs for all layers across steps, adapting to generation parameters such as resolution and denoising schedules to optimize efficiency. Applied to OpenSora, Latte, and CogVideoX, Foresight achieves up to \latencyimprv end-to-end speedup, while maintaining video quality. The source code of Foresight is available at \href{https://github.com/STAR-Laboratory/foresight}{https://github.com/STAR-Laboratory/foresight}.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。