统一编码多模态推荐信号,让大模型更懂用户行为
UniRec: Unified Multimodal Encoding for LLM-Based Recommendations
- 用特定编码器统一文本、图像、分类与数值特征
- 通过三元组结构区分属性名、类型和值,避免语义混淆
- 层级化建模用户历史序列,适合复杂推荐场景
大型语言模型在基于文本和图像的多模态推荐中展现出潜力,但真实推荐信号远超这两类。本文将推荐特征归纳为四类:文本、图像、类别特征和数值属性,并指出其异质性给大模型理解带来的挑战,尤其体现在跨模态及模态内部——如价格、评分、时间均为数值却语义不同。此外,推荐信号具有嵌套结构:用户历史是物品序列,每个物品关联多个属性。为此,我们提出UniRec,一种面向大模型推荐的统一多模态编码器。首先,使用模态专用编码器生成一致嵌入;其次,采用包含属性名、类型和值的三元组表示,分离模式与原始输入,保留语义差异;最后,通过分层Q-Former建模用户交互的嵌套结构并保持其层次组织。在多个真实世界基准上,UniRec相较先进多模态与大模型推荐方法性能提升最高达15%,消融实验也验证了各组件的有效性。
原文摘要 · Abstract (English)
Large language models have recently shown promise for multimodal recommendation, particularly with text and image inputs. Yet real-world recommendation signals extend far beyond these modalities. To reflect this, we formalize recommendation features into four modalities: text, images, categorical features, and numerical attributes, and highlight the unique challenges this heterogeneity poses for LLMs in understanding multimodal information. In particular, these challenges arise not only across modalities but also within them, as attributes such as price, rating, and time may all be numeric yet carry distinct semantic meanings. Beyond this intra-modality ambiguity, another major challenge is the nested structure of recommendation signals, where user histories are sequences of items, each associated with multiple attributes. To address these challenges, we propose UniRec, a unified multimodal encoder for LLM-based recommendation. UniRec first employs modality-specific encoders to produce consistent embeddings across heterogeneous signals. It then adopts a triplet representation, comprising attribute name, type, and value, to separate schema from raw inputs and preserve semantic distinctions. Finally, a hierarchical Q-Former models the nested structure of user interactions while maintaining their layered organization. Across multiple real-world benchmarks, UniRec outperforms state-of-the-art multimodal and LLM-based recommenders by up to 15%, and extensive ablation studies further validate the contributions of each component.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。