用轻量级音频编码器实现音乐视听问答高精度，仅需单卡5小时训练
Qwen-MusicAVQA-7B: A Multimodal Model for Music Audio-Visual QA
- 用冻结的Whisper编码器通过线性投影融合音乐与问题音频
- 在MUSIC-AVQA上达到96.0%准确率，优于大型多模态模型26个百分点
- 全程冻结编码器，推理高效，单张A100可完成全部训练与推理
为音乐视听问答（AVQA）任务，提出轻量级多模态模型Qwen-MusicAVQA-7B。该模型将冻结的Whisper编码器通过学习的线性投影连接至Qwen2-VL-7B-Instruct。同一冻结编码器分别处理视频中的音乐轨道和语音提问，语言模型通过预训练自注意力融合视觉帧、音乐与问题音频，无需任务特定融合网络。在包含7,402个问题的可用视频测试集上，系统在三个独立训练种子下取得96.0% ± 3.9%的准确率。核心发现是下游性能取决于音频表示对局部时间信息的保留程度：在32个标记的匹配比较中，步幅池化的Whisper序列比全局池化的PANNs表示高出26个百分点，尽管后者接收更多音频且使用更大投影器。即使在同一Whisper内，降低时间分辨率在固定标记预算下也会导致类似性能损失。在相同数据与输入条件下，微调后的Qwen2.5-Omni-7B准确率为80.9%，而本模型30秒版本达95.9%。在重表述的MUSIC-AVQA-R基准的采样头部与尾部子集上，准确率仍保持在96.5%与95.6%。由于两个编码器均冻结且音乐特征已缓存，整个适配过程成本极低：完整两阶段运行约需5小时（单张A100 80GB），所有实验均可在单块GPU上完成。
原文摘要 · Abstract (English)
A common approach to adding audio to a vision-language model is to train or adapt a large omni-modal system. We show that a lightweight alternative can be highly effective for music audio-visual question answering (AVQA). Qwen-MusicAVQA-7B connects a frozen Whisper encoder to Qwen2-VL-7B-Instruct through learned linear projections. The same frozen encoder processes both the video's music track and a TTS-spoken question through separate projectors, while the language model fuses visual frames, music, and question audio through pretrained self-attention, with no task-specific fusion network. On MUSIC-AVQA, our system reaches 96.0% +/- 3.9% accuracy across three independent training seeds on the 7,402-question available-video test subset. Our central finding is that downstream accuracy tracks how much fine-grained local temporal information the audio representation preserves. In a matched 32-token comparison, a stride-pooled Whisper frame sequence outperforms a globally pooled PANNs representation expanded to the same budget by 26 percentage points, even though PANNs sees at least as much audio and uses a far larger projector. The effect is not simply sequence versus vector: within Whisper alone, reducing temporal resolution at a fixed token budget costs a comparable amount. Under matched data and inputs, fine-tuned Qwen2.5-Omni-7B reaches 80.9%, against 95.9% for our 30 s variant; because the systems differ in backbone and adaptation, this is a system-level comparison. Accuracy remains high on sampled head and tail splits of the rephrased MUSIC-AVQA-R benchmark (96.5% and 95.6%). Because both encoders stay frozen and the music features are cached, the entire adaptation is cheap to train: the complete two-stage AVQA run takes approximately 5 hours on a single A100 80GB, and every run reported here fits on that one GPU.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。