音乐视觉问答需专用多模态设计，不能套用通用模型。
Music Audio-Visual Question Answering Requires Specialized Multimodal Designs
- 针对音乐内容设计专用的时空处理架构
- 结合音乐知识可提升问答准确率
- 适合研究音乐多模态理解的学者参考
尽管近期多模态大模型在通用任务中表现优异，但音乐等专业领域仍需定制化方法。音乐音频-视觉问答（Music AVQA）面临连续、密集的音视频内容、复杂的时序动态以及对领域知识的依赖等独特挑战。通过对现有数据集与方法的系统分析，本文指出：专用输入处理、融合空间-时间特性的架构设计、以及音乐特定建模策略是该领域取得成功的关键。研究揭示了与高性能密切相关的有效设计模式，提出了融入音乐先验的具体方向，旨在为推进多模态音乐理解建立坚实基础。相关工作汇总已开源于GitHub：https://github.com/WenhaoYou1/Survey4MusicAVQA。
原文摘要 · Abstract (English)
While recent Multimodal Large Language Models exhibit impressive capabilities for general multimodal tasks, specialized domains like music necessitate tailored approaches. Music Audio-Visual Question Answering (Music AVQA) particularly underscores this, presenting unique challenges with its continuous, densely layered audio-visual content, intricate temporal dynamics, and the critical need for domain-specific knowledge. Through a systematic analysis of Music AVQA datasets and methods, this paper identifies that specialized input processing, architectures incorporating dedicated spatial-temporal designs, and music-specific modeling strategies are critical for success in this domain. Our study provides valuable insights for researchers by highlighting effective design patterns empirically linked to strong performance, proposing concrete future directions for incorporating musical priors, and aiming to establish a robust foundation for advancing multimodal musical understanding. We aim to encourage further research in this area and provide a GitHub repository of relevant works: https://github.com/WenhaoYou1/Survey4MusicAVQA.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。