用感知一致性评估音乐激发的画作,更准识别画面中的音乐相关区域。
MPJudge: Towards Perceptual Assessment of Music-Induced Paintings
- 通过调制融合将音乐特征注入视觉编码器,建模跨模态感知一致性。
- 在首个专家标注的音乐-绘画对数据集上,性能超越现有方法。
- 适合关注跨模态艺术生成与感知评估的研究者。
音乐激发的绘画是一种独特的艺术实践,即在音乐影响下创作视觉作品。评估一幅画是否忠实反映其灵感音乐,是一项具有挑战性的感知评估任务。现有方法主要依赖情绪识别模型来衡量音乐与绘画间的相似性,但此类模型引入大量噪声,并忽视情绪之外的更广泛感知线索。为此,我们提出一种新颖的音乐激发绘画评估框架,直接建模音乐与视觉艺术之间的感知一致性。我们构建了首个大规模音乐-绘画对数据集MPD,由领域专家基于感知一致性进行标注,并进一步收集成对偏好标注以处理模糊案例。基于此数据集,我们提出MPJudge模型,通过调制融合机制将音乐特征融入视觉编码器。为有效学习模糊案例,采用直接偏好优化进行训练。大量实验表明,本方法优于现有方法;定性结果也显示,模型能更准确识别画作中与音乐相关的区域。
原文摘要 · Abstract (English)
Music induced painting is a unique artistic practice, where visual artworks are created under the influence of music. Evaluating whether a painting faithfully reflects the music that inspired it poses a challenging perceptual assessment task. Existing methods primarily rely on emotion recognition models to assess the similarity between music and painting, but such models introduce considerable noise and overlook broader perceptual cues beyond emotion. To address these limitations, we propose a novel framework for music induced painting assessment that directly models perceptual coherence between music and visual art. We introduce MPD, the first large scale dataset of music painting pairs annotated by domain experts based on perceptual coherence. To better handle ambiguous cases, we further collect pairwise preference annotations. Building on this dataset, we present MPJudge, a model that integrates music features into a visual encoder via a modulation based fusion mechanism. To effectively learn from ambiguous cases, we adopt Direct Preference Optimization for training. Extensive experiments demonstrate that our method outperforms existing approaches. Qualitative results further show that our model more accurately identifies music relevant regions in paintings.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。