SANA-Streaming实现实时高清视频编辑,单卡即可流畅运行。
SANA-Streaming: Real-time Streaming Video Editing with Hybrid Diffusion Transformer

- 混合扩散变压器结构提升局部建模能力,兼顾效率
- 24帧/秒实时输出,1280×704分辨率下端到端流畅
- 适合直播、游戏等对延迟敏感的实时交互场景
实时流式视频到视频编辑对直播和游戏等交互应用至关重要,但受时间一致性与推理吞吐量双重约束,仍具挑战。本文提出SANA-Streaming,一种面向消费级GPU的高分辨率实时流式视频编辑系统-算法协同设计框架。核心创新包括:(1) 混合扩散变压器架构,在部分模块引入softmax注意力以增强局部建模能力,同时保持线性层效率;(2) 循环反向正则化策略,通过光流匹配从生成内容反推源帧,实现语义一致性,无需成对长视频标注;(3) 高效系统协同设计,结合融合的GDN核与针对NVIDIA Blackwell架构优化的混合精度量化(MPQ),最大化张量核心利用率。实测显示,系统在单张RTX 5090上实现1280×704分辨率下24帧/秒端到端编辑,其中DiT核心运行达58帧/秒。实验表明,该协同设计显著优于现有最先进方法,在时间连贯性与系统吞吐量上均取得突破。
原文摘要 · Abstract (English)
Real-time streaming video-to-video editing (V2V) is critical for interactive applications such as live broadcasting and gaming, yet it remains a formidable challenge due to the stringent requirements for temporal consistency and inference throughput. In this paper, we present SANA-Streaming, a system-algorithm co-designed framework for high-resolution, real-time streaming video editing on consumer GPUs, with the following three core designs: (1) Hybrid Diffusion Transformer architecture introduces softmax attention in part of the blocks to improve local modeling capabilities while preserving the efficiency of linear layers. (2) Cycle-Reverse Regularization is a novel training strategy that enforces semantic consistency by predicting source frames from generated content via flow matching, improving temporal consistency without requiring paired long edited videos. (3) Efficient System Co-design combines fused GDN kernels and Mixed-Precision Quantization (MPQ) optimized for the NVIDIA Blackwell (RTX 5090) architecture. By profiling real-world throughput, our MPQ maximizes Tensor Core utilization while maintaining generation quality. The resulting system achieves real-time 1280 x 704 resolution editing at 24 end-to-end FPS on a single RTX 5090 GPU, with the DiT core running at 58 FPS. Experimental results demonstrate that our co-design approach significantly outperforms existing SOTA methods in both temporal coherence and system throughput.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。