轻量级扩散模型实现1080p实时图像压缩，帧率超60FPS。
CoD-Lite: Real-Time Diffusion-Based Generative Image Compression
- 用压缩导向预训练+轻量卷积替代变压器，降低计算开销。
- 1080p下编码60帧/秒、解码42帧/秒，比特率降低85%。
- 适合需要实时生成压缩的边缘设备或视频流应用。
当前先进扩散方法通常通过扩大扩散变换器来获得强生成先验，但在实时压缩场景中，模型缩放难以泛化。本文针对两个核心问题展开探索：首先，扩散预训练是否对轻量级扩散编解码器有益？系统分析表明，面向生成的预训练在小模型规模下效果较差，而面向压缩的预训练表现更优且稳定。其次，变换器是否必不可少？研究发现，尽管全局注意力对标准生成任务至关重要，但结合蒸馏技术后，轻量卷积足以满足压缩导向扩散需求。基于此，我们构建了一种单步轻量卷积扩散编解码器，在1080p下实现60帧/秒编码与42帧/秒解码。进一步通过蒸馏和对抗学习优化，该方案在比特率降低85%的同时，保持与MS-ILLM相当的FID性能，弥合了生成式压缩与实际实时部署之间的差距。代码已开源。
原文摘要 · Abstract (English)
Recent advanced diffusion methods typically derive strong generative priors by scaling diffusion transformers. However, scaling fails to generalize when adapted for real-time compression scenarios that demand lightweight models. In this paper, we explore the design of real-time and lightweight diffusion codecs by addressing two pivotal questions. First, does diffusion pre-training benefit lightweight diffusion codecs? Through systematic analysis, we find that generation-oriented pre-training is less effective at small model scales whereas compression-oriented pre-training yields consistently better performance. Second, are transformers essential? We find that while global attention is crucial for standard generation, lightweight convolutions suffice for compression-oriented diffusion when paired with distillation. Guided by these findings, we establish a one-step lightweight convolution diffusion codec that achieves real-time $60$~FPS encoding and $42$~FPS decoding at 1080p. Further enhanced by distillation and adversarial learning, the proposed codec reduces bitrate by 85\% at a comparable FID to MS-ILLM, bridging the gap between generative compression and practical real-time deployment. Codes are released at https://github.com/microsoft/GenCodec/tree/main/CoD_Lite
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。