arXiv:2511.04317cs.CV2025-11被引 3

用大模型实时重写提示词,让文本生成视频更准更稳。

RISE-T2V: Rephrasing and Injecting Semantics with LLM for Expansive Text-to-Video Generation

  • 把提示词重写和语义提取合为一步,直接用大模型隐式改写输入
  • 在多个视频扩散模型上提升生成质量,尤其对简短提示更有效
  • 适合想用简洁描述生成高质量视频的用户,可适配多种模型

大多数文本到视频(T2V)扩散模型依赖预训练文本编码器进行语义对齐,但在使用简短提示时往往难以保持视频质量,主要原因是其对文本语义的理解能力有限。此外,这些编码器无法在线重写提示以更好地匹配用户意图,限制了模型的扩展性和可用性。为此,我们提出 RISE-T2V,将提示词重写与语义特征提取融合为单一无缝步骤,而非两个独立流程。该框架具有通用性,可适配多种预训练大语言模型(LLM)和视频扩散模型(VDM),显著提升其在 T2V 任务中的表现。我们设计了名为 Re-phrasing Adapter 的创新模块,使扩散模型能利用大模型在下一词预测中产生的隐藏状态作为视频生成的条件。通过该模块,视频生成模型可隐式地将基础提示转化为更完整的表征,更贴近用户意图。同时,借助大模型的强大能力,视频生成模型得以完成更广泛的 T2V 任务。大量实验表明,RISE-T2V 适用于不同视频扩散模型架构,在生成高保真、意图对齐视频方面有显著提升。可视化结果见:https://rise-t2v.github.io。

原文摘要 · Abstract (English)

Most text-to-video(T2V) diffusion models depend on pre-trained text encoders for semantic alignment, yet they often fail to maintain video quality when provided with concise prompts rather than well-designed ones. The primary issue lies in their limited textual semantics understanding. Moreover, these text encoders cannot rephrase prompts online to better align with user intentions, which limits both the scalability and usability of the models, To address these challenges, we introduce RISE-T2V, which uniquely integrates the processes of prompt rephrasing and semantic feature extraction into a single and seamless step instead of two separate steps. RISE-T2V is universal and can be applied to various pre-trained LLMs and video diffusion models(VDMs), significantly enhancing their capabilities for T2V tasks. We propose an innovative module called the Rephrasing Adapter, enabling diffusion models to utilize text hidden states during the next token prediction of the LLM as a condition for video generation. By employing a Rephrasing Adapter, the video generation model can implicitly rephrase basic prompts into more comprehensive representations that better match the user's intent. Furthermore, we leverage the powerful capabilities of LLMs to enable video generation models to accomplish a broader range of T2V tasks. Extensive experiments demonstrate that RISE-T2V is a versatile framework applicable to different video diffusion model architectures, significantly enhancing the ability of T2V models to generate high-quality videos that align with user intent. Visual results are available on the webpage at https://rise-t2v.github.io.

文本生成视频大模型提示重写扩散模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。