让语言模型学会听声音，用检索增强理解音频常识。
AudioBERT: Audio Knowledge Augmented Language Model
- 通过检索外部音频知识，动态补充语言模型的听觉认知
- 在新构建的AuditoryBench上显著提升模型对声音常识的理解能力
- 适合研究多模态知识融合与语音感知的AI从业者
近期研究表明，仅基于文本预训练的语言模型往往缺乏基本的视觉常识，例如日常物体的颜色。受此启发，我们探究语言模型是否也存在听觉知识缺失的问题。为此，我们构建了名为AuditoryBench的新数据集，包含两项评估听觉知识的新任务。基于该基准的分析发现，语言模型同样严重缺乏听觉知识。为解决这一问题，我们提出AudioBERT，一种通过检索式方法增强BERT听觉知识的新方法：首先在提示中识别听觉知识片段并高效查询检索模型；随后将音频知识注入BERT，并在需要时启用低秩适配以实现有效微调。实验表明，AudioBERT在AuditoryBench上表现优异，显著优于基线模型。代码与数据集已开源。
原文摘要 · Abstract (English)
Recent studies have identified that language models, pretrained on text-only datasets, often lack elementary visual knowledge, \textit{e.g.,} colors of everyday objects. Motivated by this observation, we ask whether a similar shortcoming exists in terms of the \textit{auditory} knowledge. To answer this question, we construct a new dataset called AuditoryBench, which consists of two novel tasks for evaluating auditory knowledge. Based on our analysis using the benchmark, we find that language models also suffer from a severe lack of auditory knowledge. To address this limitation, we propose AudioBERT, a novel method to augment the auditory knowledge of BERT through a retrieval-based approach. First, we detect auditory knowledge spans in prompts to query our retrieval model efficiently. Then, we inject audio knowledge into BERT and switch on low-rank adaptation for effective adaptation when audio knowledge is required. Our experiments demonstrate that AudioBERT is quite effective, achieving superior performance on the AuditoryBench. The dataset and code are available at \bulurl{https://github.com/HJ-Ok/AudioBERT}.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。