用多模态模型同时解决穿搭搭配与互补商品推荐问题
A Hybrid Multimodal Deep Learning Framework for Intelligent Fashion Recommendation
- 结合CLIP视觉文本编码器，用Transformer融合图文特征
- 搭配预测AUC达0.95，补全商品检索准确率69.24%
- 适合做时尚推荐系统研究或工业落地的开发者参考
线上时尚平台的快速发展催生了对智能推荐系统的需求，要求能够理解视觉和文本信息。本文提出一种混合多模态深度学习框架，同时解决穿搭搭配预测与互补商品检索两大任务。模型利用CLIP架构的视觉与文本编码器获取时尚商品的联合潜在表示，整合为统一特征向量后输入Transformer编码器。针对搭配预测，引入“穿搭标记”建模物品间整体关系，在Polyvore数据集上AUC达到0.95；针对互补商品检索，使用“目标商品标记”表示所需商品描述，以填空（FITB）指标衡量，准确率达69.24%。该方法在两项任务上均表现优异，验证了多模态学习在时尚推荐中的有效性。
原文摘要 · Abstract (English)
The rapid expansion of online fashion platforms has created an increasing demand for intelligent recommender systems capable of understanding both visual and textual cues. This paper proposes a hybrid multimodal deep learning framework for fashion recommendation that jointly addresses two key tasks: outfit compatibility prediction and complementary item retrieval. The model leverages the visual and textual encoders of the CLIP architecture to obtain joint latent representations of fashion items, which are then integrated into a unified feature vector and processed by a transformer encoder. For compatibility prediction, an "outfit token" is introduced to model the holistic relationships among items, achieving an AUC of 0.95 on the Polyvore dataset. For complementary item retrieval, a "target item token" representing the desired item description is used to retrieve compatible items, reaching an accuracy of 69.24% under the Fill-in-the-Blank (FITB) metric. The proposed approach demonstrates strong performance across both tasks, highlighting the effectiveness of multimodal learning for fashion recommendation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。