用数字艺术生成音乐，让画作能‘奏响’。
Art2Mus: Bridging Visual Arts and Music through Cross-Modal Generation
- 基于AudioLDM2架构，融合图像与音乐跨模态生成。
- 在自建数据集上实现复杂艺术图与音乐的精准匹配。
- 适合多媒体艺术创作与AI创意工具开发者使用。
人工智能与生成模型已革新音乐创作，许多模型通过文本或视觉提示进行引导。然而，现有图像到音乐的模型仅限于简单图像，无法从复杂的数字化艺术品中生成音乐。为弥补这一空白，我们提出$ A\textit{rt2}\u001cM\textit{us}$，一种新型模型，可基于数字化艺术作品或文本输入生成音乐。该模型扩展了AudioLDM~2架构（一种文本到音频模型），并采用通过ImageBind构建的新数据集，将数字化艺术作品与音乐配对。实验表明，$ A\textit{rt2}\u001cM\textit{us}$能生成与输入刺激相契合的音乐。这些发现表明其在多媒体艺术、互动装置及AI驱动创意工具中具有广阔应用前景。
原文摘要 · Abstract (English)
Artificial Intelligence and generative models have revolutionized music creation, with many models leveraging textual or visual prompts for guidance. However, existing image-to-music models are limited to simple images, lacking the capability to generate music from complex digitized artworks. To address this gap, we introduce $\mathcal{A}\textit{rt2}\mathcal{M}\textit{us}$, a novel model designed to create music from digitized artworks or text inputs. $\mathcal{A}\textit{rt2}\mathcal{M}\textit{us}$ extends the AudioLDM~2 architecture, a text-to-audio model, and employs our newly curated datasets, created via ImageBind, which pair digitized artworks with music. Experimental results demonstrate that $\mathcal{A}\textit{rt2}\mathcal{M}\textit{us}$ can generate music that resonates with the input stimuli. These findings suggest promising applications in multimedia art, interactive installations, and AI-driven creative tools.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。