arXiv:2512.05103cs.LGcs.AI2025-12被引 3

让视频生成像写故事一样,边想边画,更准更可控。

TV2TV: A Unified Framework for Interleaved Language and Video Generation

  • 把视频生成拆成文字和画面交替生成,用混合变压器统一建模。
  • 在游戏视频上,生成质量与提示词对齐度显著提升,控制更灵活。
  • 适合需要精细剧情控制的视频创作,如游戏演示、影视分镜。

视频生成模型发展迅速,但仍难以应对需复杂语义分支或反复高层推理的复杂输出。本文提出一种新型全模态视频-文本模型TV2TV,融合大语言模型的推理能力,将视频生成分解为交错的文字与画面生成过程。TV2TV采用混合变压器(MoT)架构,联合学习语言建模(下一个词预测)与视频流匹配(下帧预测)。推理时,模型自主决定何时切换生成文字或画面,先用文字“思考”后续内容,再以像素“执行”生成。该设计将决策责任转移至语言模块,显著提升视觉质量与提示词对齐性,并支持任意时刻通过文本干预调整生成轨迹。在游戏数据上的控制实验表明,该模型在视觉质量与可控性上均有显著提升。此外,通过视觉语言模型(VLMs)为体育视频添加交错的自然语言动作描述,扩展训练数据后,模型仍保持高质量生成与强提示对齐,展现其对真实世界复杂动作序列的推理与生成能力。这些结果表明,TV2TV是实现开放式文本推理与控制的视频生成的重要进展。

原文摘要 · Abstract (English)

Video generation models are rapidly advancing, but can still struggle with complex video outputs that require significant semantic branching or repeated high-level reasoning about what should happen next. In this paper, we introduce a new class of omni video-text models that integrate ideas from recent LM reasoning advances to address this challenge. More specifically, we present TV2TV, a unified generative modeling framework which decomposes video generation into an interleaved text and video generation process. TV2TV jointly learns language modeling (next-token prediction) and video flow matching (next-frame prediction) using a Mixture-of-Transformers (MoT) architecture. At inference time, TV2TV decides when to alternate between generating text and video frames, allowing the model to "think in words" about subsequent content before ``acting in pixels'' to produce frames. This design offloads much of the responsibility for deciding what should happen next to the language modeling tower, enabling improved visual quality and prompt alignment of generated videos. It also enables fine-grained controllability, allowing users to modify the video generation trajectory through text interventions at any point in the process. In controlled experiments on video game data, TV2TV demonstrates substantial improvements in both visual quality and controllability. TV2TV also scales to natural videos, as we show by augmenting sports videos with interleaved natural language action descriptions using vision-language models (VLMs). Training TV2TV on this corpus yields strong visual quality and prompt alignment, showcasing the model's ability to reason about and generate complex real-world action sequences. Together, these results highlight TV2TV as a promising step toward video generation with open-ended textual reasoning and control.

视频生成文本控制多模态

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。