arXiv:2411.02992cs.IRcs.CV2024-11中稿 · IEEE Transactions …被引 19

IISAN-Versa让多模态大模型在推荐系统中更高效适配,支持不同大小的文本模型。

Efficient and Effective Adaptation of Multimodal Foundation Models in Sequential Recommendation

  • 采用解耦式参数高效微调,支持对称与非对称多模态模型
  • 在Microlens数据集上达到当前最优性能,且适配更大文本模型效果更好
  • 可直接插入现有系统,适合想快速部署大模型推荐的开发者

多模态基础模型(MFMs)通过先进的表示学习革新了序列推荐系统。尽管参数高效微调(PEFT)被广泛用于适配这些模型,但现有研究多关注参数效率,忽视了显存占用和训练速度。为此,我们提出IISAN框架以显著提升效率,但其仅适用于对称型MFMs及相同文本与图像编码器,限制了先进大语言模型的应用。为突破此局限,我们开发了IISAN-Versa,一种兼容对称与非对称结构的通用即插即用架构。该方法采用解耦式PEFT,结合模内与模间适配机制,通过分组层丢弃与维度对齐变换有效处理异构性。实验表明,IISAN-Versa能有效适配大型文本编码器,并发现编码器越大性能越优的缩放效应。它在包含原始标题及图像/视频生成字幕的多种多模态场景中表现优异,在公开的Microlens基准上达到当前最优结果。代码已开源:https://github.com/GAIR-Lab/IISAN。

原文摘要 · Abstract (English)

Multimodal foundation models (MFMs) have revolutionized sequential recommender systems through advanced representation learning. While Parameter-efficient Fine-tuning (PEFT) is commonly used to adapt these models, studies often prioritize parameter efficiency, neglecting GPU memory and training speed. To address this, we introduced the IISAN framework, significantly enhancing efficiency. However, IISAN was limited to symmetrical MFMs and identical text and image encoders, preventing the use of state-of-the-art Large Language Models. To overcome this, we developed IISAN-Versa, a versatile plug-and-play architecture compatible with both symmetrical and asymmetrical MFMs. IISAN-Versa employs a Decoupled PEFT structure and utilizes both intra- and inter-modal adaptation. It effectively handles asymmetry through a simple yet effective combination of group layer-dropping and dimension transformation alignment. Our research demonstrates that IISAN-Versa effectively adapts large text encoders, and we further identify a scaling effect where larger encoders generally perform better. IISAN-Versa also demonstrates strong versatility in our defined multimodal scenarios, which include raw titles and captions generated from images and videos. Additionally, IISAN-Versa achieved state-of-the-art performance on the Microlens public benchmark. We release our code at https://github.com/GAIR-Lab/IISAN.

推荐系统多模态参数高效大模型适配

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。