arXiv:2607.13471cs.CVcs.MM2026-07

根据音乐情绪生成沉浸式360°视频,让听觉体验可视化。

Bring Music The Horizon: Music-Driven 360$^\circ$ Video Generation

论文配图:Bring Music The Horizon: Music-Driven 360$^\circ$ Video Generation
图 1 · 摘自论文原文
  • 通过四小节为单位预测情绪变化,生成情感引导信号
  • 利用SEGA框架实现关键帧的细粒度语义控制
  • 适合音乐可视化、沉浸式内容创作的开发者

音乐可视化通过将听觉信号转化为视觉形式,有效增强听众对音乐的理解与体验。然而,现有方法大多依赖歌词,或生成平面、非沉浸式视频,难以传达音乐的情感动态与沉浸感。本文提出Bring Music The Horizon,一个情感感知的音乐驱动360°视频生成流程。给定一首歌曲,首先以每四小节为单位预测其情绪轨迹(效价-唤醒值),再通过EmotiCrafter将这些值转化为情感引导向量,并由SEGA框架进行细粒度语义控制,生成关键帧;最后利用图像到视频模型合成时间连续的360°视频。该流程可生成反映输入歌曲情绪演变与时间结构的360°可视化视频。我们在不同音乐风格上验证了其能力,并在项目页(https://etoile-et-toi-mp3.github.io/BMTH_Project_Page/)与基线模型From-Sound-To-Sight进行了定性对比。

原文摘要 · Abstract (English)

Music visualization offers a powerful way to enhance listeners' understanding and experience of music by translating auditory signals into visual forms. However, most existing approaches either rely heavily on lyrics or generate flat, non-immersive videos similar to conventional music videos, which limits their ability to convey the emotional dynamics of music and provide an immersive listening experience. We propose Bring Music The Horizon, an emotion-aware pipeline for music-driven 360$^\circ$ video generation. Given an input song, our work first estimates its emotional trajectory by predicting valence-arousal values at the level of every four bars. These values are then converted into emotion-aware visual guidance using EmotiCrafter, and these guidance vectors can be manipulated by the SEGA framework, which provides fine-grained semantic control for keyframe generation. Finally, image-to-video models are applied to the generated keyframes to synthesize temporally continuous 360$^\circ$ videos for immersive music visualization. Our pipeline generates 360$^\circ$ music visualization videos that reflect the emotional progression and temporal structure of the input song. We demonstrate its capability using songs from different genres and provide qualitative comparisons with From-Sound-To-Sight, a representative audio-to-visual generation baseline, on our project page at https://etoile-et-toi-mp3.github.io/BMTH_Project_Page/.

音乐可视化360°视频情感生成

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。