用元学习动态融合多模态信息,提升短视频推荐效果
Dynamic Multimodal Fusion via Meta-Learning Towards Micro-Video Recommendation
- 以元学习为框架,为每条视频定制融合策略
- 在三个数据集上超越MMGCN等前沿模型
- 适合做多模态推荐与个性化内容系统的研究者
多模态信息(如视觉、音频、文本)广泛用于增强短视频推荐中的表征学习。现有方法依赖静态多模态融合,难以捕捉不同视频间多模态关系的差异。本文提出基于元学习的动态多模态融合框架MetaMMF,将每条视频的多模态融合视为独立任务。通过元学习器根据输入任务的多模态特征提取元信息,参数化神经网络生成针对特定物品的融合函数。在三个基准数据集上进行大量实验,结果表明其显著优于MMGCN、LATTICE和InvRL等先进模型。为进一步提升训练效率,采用典型阵列分解(canonical polyadic decomposition)对模型轻量化,并验证了有效性。代码已开源。
原文摘要 · Abstract (English)
Multimodal information (e.g., visual, acoustic, and textual) has been widely used to enhance representation learning for micro-video recommendation. For integrating multimodal information into a joint representation of micro-video, multimodal fusion plays a vital role in the existing micro-video recommendation approaches. However, the static multimodal fusion used in previous studies is insufficient to model the various relationships among multimodal information of different micro-videos. In this paper, we develop a novel meta-learning-based multimodal fusion framework called Meta Multimodal Fusion (MetaMMF), which dynamically assigns parameters to the multimodal fusion function for each micro-video during its representation learning. Specifically, MetaMMF regards the multimodal fusion of each micro-video as an independent task. Based on the meta information extracted from the multimodal features of the input task, MetaMMF parameterizes a neural network as the item-specific fusion function via a meta learner. We perform extensive experiments on three benchmark datasets, demonstrating the significant improvements over several state-of-the-art multimodal recommendation models, like MMGCN, LATTICE, and InvRL. Furthermore, we lighten our model by adopting canonical polyadic decomposition to improve the training efficiency, and validate its effectiveness through experimental results. Codes are available at https://github.com/hanliu95/MetaMMF.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。