提升大模型对音频时序的理解能力，增强其在设备端的应用潜力。
Enhancing Temporal Understanding in Audio Question Answering for Large Audio Language Models
- 用大模型生成可靠时序问答数据，扩充训练集
- 通过课程学习微调，显著提升时序推理能力
- 成功部署到本地设备，支持低延迟推理
音频问答（AQA）任务涵盖音频事件分类、音频描述生成和开放性推理。随着大音频语言模型（LALMs）的出现，该任务受到广泛关注。现有方法通常将音频编码器与仅文本的大语言模型（LLM）通过投影模块结合，构建LALMs。尽管这些模型在通用音频理解上表现优异，但在时序推理方面存在局限，制约了其商业应用和边缘部署。本文针对这一问题提出改进：首先，利用大模型生成高质量的音频时序问答数据，实现数据增强；其次，采用课程学习策略对现有基线模型进行微调，使其在保持原有任务性能的同时，显著提升时序推理能力；最后，我们在公共音频基准数据集上验证了模型性能，并实现了模型在本地设备上的部署，测试了其在CPU上的推理效率，为边缘应用场景提供可行方案。
原文摘要 · Abstract (English)
The Audio Question Answering (AQA) task includes audio event classification, audio captioning, and open-ended reasoning. Recently, AQA has garnered attention due to the advent of Large Audio Language Models (LALMs). Current literature focuses on constructing LALMs by integrating audio encoders with text-only Large Language Models (LLMs) through a projection module. While LALMs excel in general audio understanding, they are limited in temporal reasoning, which may hinder their commercial applications and on-device deployment. This paper addresses these challenges and limitations in audio temporal reasoning. First, we introduce a data augmentation technique for generating reliable audio temporal questions and answers using an LLM. Second, we perform a further fine-tuning of an existing baseline using curriculum learning strategy to specialize in temporal reasoning without compromising performance on fine-tuned tasks. We demonstrate the performance of our model using state-of-the-art LALMs on public audio benchmark datasets. Third, we implement our AQA model on-device locally and investigate its CPU inference for edge applications.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。