arXiv:2412.17171cs.LGcs.IR2024-12被引 16

让大模型自己优化物品分词,提升推荐精度。

Enhancing Item Tokenization for Generative Recommendation through Self-Improvement

  • 大模型在训练中自我调整物品分词方式,与自身理解对齐。
  • 实验显示推荐性能平均提升8%,跨数据集表现稳定。
  • 无需复杂改造,可直接接入现有推荐系统使用。

生成式推荐系统借助大语言模型(LLM)将物品建模为标记序列,以生成方式预测用户偏好。其关键挑战在于物品的有效分词,需适配LLM的标记化机制。现有方法包括文本描述、数值字符串或自定义离散标记序列。文本表示虽兼容性强,但过长导致效率低下;数值字符串简洁却缺乏语义;自定义标记序列需外部模型分配,易与LLM内部预训练分词不一致,影响性能。为此,我们提出一种自改进的物品分词方法,使LLM在训练中自主优化其初始分词结果。该方法基于任意外部模型生成的初始分词,通过学习模式定期修正分词策略,确保分词与模型内部理解一致,从而提升推荐准确性。该方法实现简单,可作为即插即用模块集成至现有系统。多数据集实验及多种初始化策略验证了其有效性,平均推荐性能提升8%。

原文摘要 · Abstract (English)

Generative recommendation systems, driven by large language models (LLMs), present an innovative approach to predicting user preferences by modeling items as token sequences and generating recommendations in a generative manner. A critical challenge in this approach is the effective tokenization of items, ensuring that they are represented in a form compatible with LLMs. Current item tokenization methods include using text descriptions, numerical strings, or sequences of discrete tokens. While text-based representations integrate seamlessly with LLM tokenization, they are often too lengthy, leading to inefficiencies and complicating accurate generation. Numerical strings, while concise, lack semantic depth and fail to capture meaningful item relationships. Tokenizing items as sequences of newly defined tokens has gained traction, but it often requires external models or algorithms for token assignment. These external processes may not align with the LLM's internal pretrained tokenization schema, leading to inconsistencies and reduced model performance. To address these limitations, we propose a self-improving item tokenization method that allows the LLM to refine its own item tokenizations during training process. Our approach starts with item tokenizations generated by any external model and periodically adjusts these tokenizations based on the LLM's learned patterns. Such alignment process ensures consistency between the tokenization and the LLM's internal understanding of the items, leading to more accurate recommendations. Furthermore, our method is simple to implement and can be integrated as a plug-and-play enhancement into existing generative recommendation systems. Experimental results on multiple datasets and using various initial tokenization strategies demonstrate the effectiveness of our method, with an average improvement of 8\% in recommendation performance.

生成推荐大模型分词优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。