用图文描述提升餐厅推荐效果，验证多模态信息有用。
Multimodal Point-of-Interest Recommendation
- 将图片转为文本描述，构建半多模态推荐数据集。
- 模型在含图文描述的数据上准确率显著更高。
- 适合做基于视觉与语言的推荐系统研究者参考。
大语言模型已应用于购物和新闻推荐。点位推荐（Point of Interest）是基于语言表示的序列推荐新领域。本文以餐厅推荐为例，研究用户根据历史到访记录选择下一家餐厅时，会考虑菜系、位置及菜品图片。我们利用Foursquare和FoodX-251数据集，通过多模态模型LLaVA将图片转换为文本描述，构建伪餐厅打卡历史数据集，并采用2023年提出的Recformer语言序列推荐框架。在包含图文描述的半多模态数据集上训练的模型，优于仅使用纯文本数据的模型，表明该半多模态方法能反映真实人类行为，验证了多模态推荐路径的可行性。
原文摘要 · Abstract (English)
Large Language Models are applied to recommendation tasks such as items to buy and news articles to read. Point of Interest is quite a new area to sequential recommendation based on language representations of multimodal datasets. As a first step to prove our concepts, we focused on restaurant recommendation based on each user's past visit history. When choosing a next restaurant to visit, a user would consider genre and location of the venue and, if available, pictures of dishes served there. We created a pseudo restaurant check-in history dataset from the Foursquare dataset and the FoodX-251 dataset by converting pictures into text descriptions with a multimodal model called LLaVA, and used a language-based sequential recommendation framework named Recformer proposed in 2023. A model trained on this semi-multimodal dataset has outperformed another model trained on the same dataset without picture descriptions. This suggests that this semi-multimodal model reflects actual human behaviours and that our path to a multimodal recommendation model is in the right direction.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。