arXiv:2605.25659cs.CV2026-05被引 2

StreamChar实现角色音视频流式生成,实时稳定且保持视觉一致。

StreamChar: Long-Horizon Streaming Character Audio-Video Generation with Decoupled Orchestration

论文配图:StreamChar: Long-Horizon Streaming Character Audio-Video Generation with Decoupled Orchestration
图 1 · 摘自论文原文
  • 分步设计:用LLM orchestrator规划音频,DiT模型局部去噪生成
  • 单卡H100实时运行,长时生成仍保音画同步与画面质量
  • 适合需要低延迟、高一致性的虚拟角色动画应用

角色动画的实时流式音视频生成需满足:准确朗读指定文本、跨片段保持视觉一致性、在严格播放预算内运行。传统方法中,逐段自回归生成易导致语音与文本错位及视觉漂移,而为降低延迟采用的少步蒸馏又会削弱空间多样性与时序质量。本文提出StreamChar,将长时序编排与短窗口音视频去噪解耦:基于LLM的编排器结合文本和历史信息生成帧对齐的音频条件;联合音视频DiT模型在参考帧与运动帧条件下进行局部双向去噪。为高效部署,采用两阶段蒸馏:先压缩采样器,再在在线分块滚动训练中微调学生模型。进度感知指针确保部分文本与生成音频对齐,沉降块内存提供持久视觉锚点以减少长时漂移。在短片段与长时序任务上的实验表明,StreamChar可在单张H100 GPU上实时运行,在文本准确性、音画同步性、视觉质量与流式稳定性之间取得良好系统级权衡,优于近期联合与音频驱动基线方法。

原文摘要 · Abstract (English)

Real-time streaming joint audio-video generation for character animation requires a generator to speak the requested transcript, maintain visual identity across chunks, and run within a strict playback budget. These requirements are difficult to satisfy simultaneously: chunk-wise autoregressive generation can accumulate transcript-audio misalignment and visual drift, while the few-step distillation needed for low latency often degrades spatial diversity and temporal quality. We present StreamChar, a streaming framework that separates long-horizon orchestration from short-window audio-video denoising. An LLM-based orchestrator uses the transcript and historical context to produce frame-aligned audio conditions, and a joint audio-video DiT performs local bidirectional denoising with reference and motion-frame conditioning. For efficient deployment, we use a two-stage distillation pipeline that first compresses the sampler and then fine-tunes the student under online chunk rollouts. A progress-aware pointer aligns partial transcripts with generated audio during rollout training, and a sink-chunk memory provides a persistent visual anchor for reducing long-horizon drift. Experiments on short-clip and long-horizon protocols show that StreamChar runs in real time on a single H100 GPU and provides a favorable system-level trade-off among transcript fidelity, audio-visual synchronization, visual quality, and streaming stability compared with recent joint and audio-driven baselines.

音视频生成流式生成角色动画扩散模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。