arXiv:2504.12007cs.IRcs.AI2025-04中稿 · The ACM Web Confer…被引 12

用连续令牌提升大模型推荐系统,生成更精准的用户偏好。

Diffusion Generative Recommendation with Continuous Tokens

  • 引入连续令牌编码用户/物品,避免离散量化损失
  • 通过扩散模型生成高质量用户偏好,效果优于现有方法
  • 适合关注大模型与推荐系统融合的研究者

近年来,生成式人工智能特别是大语言模型(LLMs)的发展为推荐系统(RecSys)带来了新机遇。现有基于LLM的推荐方法多在离散空间中运行,依赖向量量化分词器以适配语言模型的离散特性,但此类量化常导致信息丢失和学习次优,主要源于标准向量量化中非可微argmin操作引起的梯度传播不准。受语言模型中连续令牌趋势启发,我们提出ContRec,一个将连续令牌无缝融入LLM-based RecSys的新框架。ContRec包含两个核心模块:一个用于编码用户/物品的sigma-VAE分词器,采用连续变分自编码器目标训练,并结合三种技术防止表示崩溃;以及一个发散扩散模块,通过条件扩散过程捕捉隐含用户偏好。该模块在生成过程中基于大模型前序输出进行条件扩散,采用新型发散损失,实现高质量用户偏好生成。最终,ContRec结合大模型的文本推理输出与扩散模型的潜在表征,完成Top-K物品检索,提供全面推荐结果。在四个数据集上的大量实验表明,ContRec持续优于传统及当前最先进方法。结果凸显了连续分词与生成建模对下一代推荐系统发展的潜力。

原文摘要 · Abstract (English)

Recent advances in generative artificial intelligence, particularly large language models (LLMs), have opened new opportunities for enhancing recommender systems (RecSys). Most existing LLM-based RecSys approaches operate in a discrete space, using vector-quantized tokenizers to align with the inherent discrete nature of language models. However, these quantization methods often result in lossy tokenization and suboptimal learning, primarily due to inaccurate gradient propagation caused by the non-differentiable argmin operation in standard vector quantization. Inspired by the emerging trend of embracing continuous tokens in language models, we propose ContRec, a novel framework that seamlessly integrates continuous tokens into LLM-based RecSys. Specifically, ContRec consists of two key modules: a sigma-VAE Tokenizer, which encodes users/items with continuous tokens; and a Dispersive Diffusion module, which captures implicit user preference. The tokenizer is trained with a continuous Variational Auto-Encoder (VAE) objective, where three effective techniques are adopted to avoid representation collapse. By conditioning on the previously generated tokens of the LLM backbone during user modeling, the Dispersive Diffusion module performs a conditional diffusion process with a novel Dispersive Loss, enabling high-quality user preference generation through next-token diffusion. Finally, ContRec leverages both the textual reasoning output from the LLM and the latent representations produced by the diffusion model for Top-K item retrieval, thereby delivering comprehensive recommendation results. Extensive experiments on four datasets demonstrate that ContRec consistently outperforms both traditional and SOTA LLM-based recommender systems. Our results highlight the potential of continuous tokenization and generative modeling for advancing the next generation of recommender systems.

推荐系统扩散模型连续令牌大模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。