通过稀疏学习提升音乐演出问答的效率与准确率
Learning Sparsity for Effective and Efficient Music Performance Question Answering
- 引入端到端稀疏框架Sparsify，融合三种稀疏化策略
- 训练时间减少28.32%，仅用25%数据仍保持70-80%性能
- 适合追求高效推理与小样本应用的多模态研究者
音乐表演以密集连续的音频和无缝音视频融合为特征，给多模态场景理解与推理带来独特挑战。近期提出的音乐表演音视频问答（Music AVQA）数据集反映了这些挑战，凸显了在复杂问答中更有效整合音视频表征的持续需求。然而，现有Music AVQA方法常依赖密集且未优化的表征，导致关键信息提取不充分、冗余未消除、重要样本难聚焦。为此，我们提出Sparsify——一种专为Music AVQA设计的稀疏学习框架，将三种稀疏化策略集成于端到端流程，在Music AVQA数据集上达到领先性能。同时，相比全量训练的稠密模型，其训练时间减少28.32%且保持精度。为进一步提升数据效率，我们提出关键子集选择算法，仅使用约25%的MUSIC-AVQA v2.0训练数据，即可在不同模型上保留70%-80%的全数据性能。
原文摘要 · Abstract (English)
Music performances, characterized by dense and continuous audio as well as seamless audio-visual integration, present unique challenges for multimodal scene understanding and reasoning. Recent Music Performance Audio-Visual Question Answering (Music AVQA) datasets have been proposed to reflect these challenges, highlighting the continued need for more effective integration of audio-visual representations in complex question answering. However, existing Music AVQA methods often rely on dense and unoptimized representations, leading to inefficiencies in the isolation of key information, the reduction of redundancy, and the prioritization of critical samples. To address these challenges, we introduce Sparsify, a sparse learning framework specifically designed for Music AVQA. It integrates three sparsification strategies into an end-to-end pipeline and achieves state-of-the-art performance on the Music AVQA datasets. In addition, it reduces training time by 28.32% compared to its fully trained dense counterpart while maintaining accuracy, demonstrating clear efficiency gains. To further improve data efficiency, we propose a key-subset selection algorithm that selects and uses approximately 25% of MUSIC-AVQA v2.0 training data and retains 70-80% of full-data performance across models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。