ReGenVC用极低码率实时生成对话头视频，比传统编码快10倍且更清晰。
ReGenVC: End-to-End Real-Time Generative Video Coding at Ultra-Low Bitrate
- 用神经压缩首帧+姿态关键点+元数据，77帧仅26kB
- 在26kB下保持画质锐利，传统编码则出现块状伪影
- 四步蒸馏+三技术优化，8卡实现实时24帧输出
我们提出ReGenVC，一种端到端的生成式视频编码器，可将对话头视频压缩至超低码率并实现实时解码。编码器将源视频压缩为紧凑比特流——一个神经压缩的首帧、每帧姿态关键点和元数据，77帧序列总计约26 kB。解码器是四步蒸馏扩散Transformer，基于传输的姿态和参考帧重建视频。相比x264/x265，ReGenVC将码率降至约十分之一（26 kB vs. 250–280 kB），实现近乎无伪影重建；在相同超低码率下，传统编码崩溃为块状伪影，而ReGenVC凭借强大的生成先验保持清晰。部署核心障碍是解码延迟：多步采样与Transformer、VAE组件过慢。我们通过四步蒸馏及三项模型保真系统技术实现实时性：(i) 八GPU统一序列并行（Ulysses & Ring），(ii) 空间分割的VAE，(iii) 三阶段重叠流水线；分析时间模型界定实时可行性区域。在8-GPU节点上，系统维持24 fps输出（每25帧窗口972毫秒，低于1000毫秒预算），支持浏览器直播无帧丢失。混合CPU-GPU部署中，编码器在CPU以24 fps运行，解码侧单次条件编码器移至CPU，每卡内存峰值从21.1 GB降至约7.7 GB。据我们所知，ReGenVC是首个在8-GPU系统上结合超低码率编码与实时解码的端到端生成式视频编码器。
原文摘要 · Abstract (English)
We present ReGenVC, an end-to-end generative video codec that compresses talking-head video to an ultra-low bitrate and decodes it in real time. The encoder reduces a source clip to a compact bitstream -- a neurally compressed first frame, per-frame pose keypoints, and metadata -- totaling about 26 kB for a 77-frame sequence. The decoder is a four-step distilled diffusion transformer that reconstructs the video conditioned on the transmitted pose and reference frame. Compared with x264/x265, ReGenVC reduces the bitrate to roughly one tenth of that required by traditional codecs (about 26 kB vs. 250--280 kB for essentially artifact-free reconstruction); at a matched ultra-low bitrate, conventional codecs collapse into blocking artifacts while ReGenVC stays sharp by exploiting a strong generative prior. The central obstacle to deploying such a codec is decoder latency: multi-step sampling with transformer and VAE components is too slow for interactive use. We make the decoder real-time through four-step distillation and three model-preserving system techniques: (i) eight-GPU unified sequence parallelism (Ulysses & Ring), (ii) a spatially-split VAE, and (iii) a three-stage overlapped pipeline; an analytical timing model characterizes the real-time feasibility region. On an 8-GPU node, the system sustains 24 fps output (972 ms per 25-frame window, within the 1000 ms budget), enabling a live browser stream without observed frame underruns. A hybrid CPU-GPU deployment further runs the encoder on the CPU at 24 fps and offloads the decoder-side one-shot conditioning encoders to the CPU, reducing the per-GPU memory peak from 21.1 GB to about 7.7 GB. To our knowledge, ReGenVC is the first end-to-end generative video codec to combine ultra-low-bitrate encoding with real-time decoding on an 8-GPU system.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。