arXiv:2604.13509cs.CV2026-04被引 2

用自回归DiT实现流畅实时视频风格化,支持动态切换提示与参考图。

DiT as Real-Time Rerenderer: Streaming Video Stylization with Autoregressive Diffusion Transformer

论文配图:DiT as Real-Time Rerenderer: Streaming Video Stylization with Autoregressive Diffusion Transformer
图 1 · 摘自论文原文
  • 基于DiT构建自回归模型,通过教师蒸馏加速生成过程。
  • 支持长视频稳定处理,实测在10秒内完成128帧风格化输出。
  • 可实时切换文本提示或参考图像,适合交互式创作场景。

近期视频生成模型的发展显著提升了视频生成及相关下游任务的效率。其中,视频风格化在沉浸式应用和艺术创作中具有重要价值,备受关注。然而,现有基于扩散模型的视频风格化方法在处理长视频时难以保持稳定性与一致性,且计算开销大、需多步去噪,限制了实际应用。本文提出RTR-DiT(DiT作为实时重渲染器),一种基于Diffusion Transformer的流式视频风格化框架。首先在精心构建的视频风格化数据集上微调双向教师模型,支持文本引导和参考图像引导的风格化任务;随后通过自强化与分布匹配蒸馏,将教师模型压缩为少步自回归模型。此外,我们提出一种参考保留的键值缓存更新策略,不仅保证长视频处理的稳定一致,还支持文本提示与参考图像之间的实时切换。实验结果表明,RTR-DiT在定量指标与视觉质量上均优于现有方法,在实时长视频风格化与交互式风格切换应用中表现优异。

原文摘要 · Abstract (English)

Recent advances in video generation models has significantly accelerated video generation and related downstream tasks. Among these, video stylization holds important research value in areas such as immersive applications and artistic creation, attracting widespread attention. However, existing diffusion-based video stylization methods struggle to maintain stability and consistency when processing long videos, and their high computational cost and multi-step denoising make them difficult to apply in practical scenarios. In this work, we propose RTR-DiT (DiT as Real-Time Rerenderer), a steaming video stylization framework built upon Diffusion Transformer. We first fine-tune a bidirectional teacher model on a curated video stylization dataset, supporting both text-guided and reference-guided video stylization tasks, and subsequently distill it into a few-step autoregressive model via post-training with Self Forcing and Distribution Matching Distillation. Furthermore, we propose a reference-preserving KV cache update strategy that not only enables stable and consistent processing of long videos, but also supports real-time switching between text prompts and reference images. Experimental results show that RTR-DiT outperforms existing methods in both text-guided and reference-guided video stylization tasks, in terms of quantitative metrics and visual quality, and demonstrates excellent performance in real-time long video stylization and interactive style-switching applications.

视频风格化扩散模型自回归生成实时渲染

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。