用大模型为默片生成高质量配乐,支持多维度可控创作。
FilmComposer: LLM-Driven Music Production for Silent Film Clips
- 构建多智能体系统,融合波形与符号音乐生成,模拟专业作曲流程。
- 在7418段默片上实现音画一致、音乐性与发展的综合提升,性能达当前最优。
- 支持节奏、语义、视觉等控制,适合影视音乐创作者交互使用。
本文提出FilmComposer,一种基于大语言模型的默片配乐生成方法。针对电影音乐制作的专业需求,该框架首次结合大规模生成模型与多智能体协作,融合波形与符号音乐生成优势。它聚焦音频质量、音乐性与音乐发展三大核心要素,引入节奏、语义和视觉等多维控制机制。系统包含视觉处理模块、可控节奏的MusicGen及多智能体评估、编排与混音模块,可无缝嵌入实际制作流程并支持用户全程干预,具备高交互性与创作自由度。为弥补高质量电影音乐数据集缺失,我们构建了MusicPro-7k数据集,含7,418个默片片段、对应音乐、描述、节奏点与主旋律。标准指标与新设计的专项指标均表明,生成音乐在质量、音画一致性、多样性、音乐性与音乐发展方面达到当前最优水平。
原文摘要 · Abstract (English)
In this work, we implement music production for silent film clips using LLM-driven method. Given the strong professional demands of film music production, we propose the FilmComposer, simulating the actual workflows of professional musicians. FilmComposer is the first to combine large generative models with a multi-agent approach, leveraging the advantages of both waveform music and symbolic music generation. Additionally, FilmComposer is the first to focus on the three core elements of music production for film-audio quality, musicality, and musical development-and introduces various controls, such as rhythm, semantics, and visuals, to enhance these key aspects. Specifically, FilmComposer consists of the visual processing module, rhythm-controllable MusicGen, and multi-agent assessment, arrangement and mix. In addition, our framework can seamlessly integrate into the actual music production pipeline and allows user intervention in every step, providing strong interactivity and a high degree of creative freedom. Furthermore, we propose MusicPro-7k which includes 7,418 film clips, music, description, rhythm spots and main melody, considering the lack of a professional and high-quality film music dataset. Finally, both the standard metrics and the new specialized metrics we propose demonstrate that the music generated by our model achieves state-of-the-art performance in terms of quality, consistency with video, diversity, musicality, and musical development. Project page: https://apple-jun.github.io/FilmComposer.github.io/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。