让自动配音更真实,模拟导演与演员的互动指导过程。
Towards Authentic Movie Dubbing with Retrieve-Augmented Director-Actor Interaction Learning
- 构建多模态参考视频库,用大模型理解情感线索。
- 根据情感相似性检索相关片段,辅助演员内化情绪。
- 分阶段融合情感信息,生成更具表现力的配音。
自动电影配音模型从脚本生成生动语音,仅凭简短音色提示即可复现说话人音色,并与无声视频实现口型同步。现有方法简化流程,忽略导演与演员间的协作关键环节。真实流程中,导演会引导演员先理解情感等上下文信息再表演。为此,我们提出一种检索增强的导演-演员交互学习方案(Authentic-Dubber),包含三项创新:(1) 构建多模态参考视频库,模拟导演提供的学习素材;利用大语言模型(LLMs)深入理解跨模态信号中的情感表征。(2) 提出基于情感相似性的检索增强策略,从库中获取与目标视频最匹配的多模态信息,帮助演员高效全面地内化情绪。(3) 设计渐进式图结构语音生成方法,逐步融合检索到的情感知识,模拟演员最终配音过程。该方案忠实还原真实配音流程,在V2C Animation基准数据集上的主客观评估均验证其有效性。代码与演示已公开于https://github.com/AI-S2-Lab/Authentic-Dubber。
原文摘要 · Abstract (English)
The automatic movie dubbing model generates vivid speech from given scripts, replicating a speaker's timbre from a brief timbre prompt while ensuring lip-sync with the silent video. Existing approaches simulate a simplified workflow where actors dub directly without preparation, overlooking the critical director-actor interaction. In contrast, authentic workflows involve a dynamic collaboration: directors actively engage with actors, guiding them to internalize the context cues, specifically emotion, before performance. To address this issue, we propose a new Retrieve-Augmented Director-Actor Interaction Learning scheme to achieve authentic movie dubbing, termed Authentic-Dubber, which contains three novel mechanisms: (1) We construct a multimodal Reference Footage library to simulate the learning footage provided by directors. Note that we integrate Large Language Models (LLMs) to achieve deep comprehension of emotional representations across multimodal signals. (2) To emulate how actors efficiently and comprehensively internalize director-provided footage during dubbing, we propose an Emotion-Similarity-based Retrieval-Augmentation strategy. This strategy retrieves the most relevant multimodal information that aligns with the target silent video. (3) We develop a Progressive Graph-based speech generation approach that incrementally incorporates the retrieved multimodal emotional knowledge, thereby simulating the actor's final dubbing process. The above mechanisms enable the Authentic-Dubber to faithfully replicate the authentic dubbing workflow, achieving comprehensive improvements in emotional expressiveness. Both subjective and objective evaluations on the V2C Animation benchmark dataset validate the effectiveness. The code and demos are available at https://github.com/AI-S2-Lab/Authentic-Dubber.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。