arXiv:2606.00125cs.IRcs.AI2026-06

用大模型融合音轨、歌词和用户行为,提升音乐推荐准确率。

Multimodal Music Recommendation System using LLMs

论文配图:Multimodal Music Recommendation System using LLMs
图 1 · 摘自论文原文
  • 引入音频、歌词和用户行为三类信号,用大模型统一建模。
  • 相比仅用歌曲ID的基线,召回率提升95%,NDCG提升79%。
  • 适合对内容理解与多模态融合感兴趣的推荐系统研究者。

音乐推荐系统通常将歌曲视为不可解释的标识符,依赖协同交互历史,忽略了语义或声学内容。已有研究探索了大模型增强、多模态及文本增强的序列推荐方法,但尚未有工作在统一的大模型框架中联合建模语义、声学和用户参与信号,并基于实际歌曲内容生成推荐。本文提出一种会话式音乐推荐的多模态框架,对LastFM-1K数据集进行了扩展,加入了三类互补信号:(1) 使用预训练音乐与文本模型提取的音频和歌词嵌入;(2) 基于MGPHot标注模式由大模型生成的语义元数据;(3) 听歌完成率。在E4SRec框架基础上,引入多种物品ID编码器(SASRec、BERT4Rec、GRU4Rec),并扩展大模型骨干网络,采用LLaMa-2-13B、Qwen2.5-7B-Instruct和LLaMa-3-70B,在零样本与微调设置下进行实验。结果表明,融合内容特征后,召回率较仅用ID的基线最高提升95%,NDCG提升79%。此外,实验显示简单的多模态融合并不总带来增益,凸显跨模态整合的挑战。研究同时发布了一个大规模多模态音乐推荐基准数据集。

原文摘要 · Abstract (English)

Music recommendation systems typically treat songs as opaque tokens, relying on collaborative interaction histories which overlooks semantic or acoustic content. Prior work has explored LLM-augmented, multimodal, and text-enhanced approaches to sequential recommendation, and while some methods partially combine semantic, acoustic, or engagement signals, none jointly model all three within a unified LLM-based sequential reasoning framework that grounds recommendations in actual song content. In this work, we propose a multimodal framework for session-based music recommendation that enriches the LastFM-1K dataset with three complementary signals: (1) audio and lyric embeddings extracted using pretrained music and text representation models, (2) LLM-generated semantic metadata using the MGPHot annotation schema, and (3) listening completion ratios. We adopt the E4SRec framework by extending it with multimodal features and different item ID encoder backbones, including SASRec, BERT4Rec, and GRU4Rec. We further extend the LLM backbone option with LLaMa-2-13B, Qwen2.5-7B-Instruct, and LLaMa-3-70B in both zero-shot and fine-tuned settings. Our experiments show that integrating content-based features improves over ID-only baselines up to 95% in terms of Recall and 79% in terms of NDCG. Moreover, our experiments show that naive multimodal fusion does not always yield additive improvements, highlighting challenges in cross-modal integration. We release a large-scale multimodal benchmark for music recommendation.

音乐推荐多模态大模型序列建模

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。