通过语义中心初始化推荐项编码,显著提升大模型推荐效果。
Preserving Item Semantics for Free: Rethinking Token Initialization in LLM-Based Generative Recommendation
- 用语义嵌入中心直接初始化项编码,保留语义结构
- 冷启动项召回率提升60%,纯微调效率提高40%
- 无需额外训练,代码改动极小,适合快速部署
生成式推荐系统利用大语言模型作为推荐主干,通过在模型词表中加入项目语义标识符(SIDs)来表示项目。理想情况下,这些标识符应赋予项嵌入以语义先验,提升模型泛化能力。然而,传统方法将新词元初始化为随机高斯向量,丢失了语义空间的连续结构,迫使模型从交互数据中重新学习项间关系。我们发现,这种初始化导致SID嵌入围绕项目流行度组织而非语义。尽管持续预训练(CPT)部分缓解该问题,但无法可靠恢复原始语义结构。为此,我们提出一种参数无成本的干预:直接用语义嵌入空间中的中心点初始化SID。仅需少量代码修改,无额外训练或推理开销。实验表明,该方法在纯监督微调下,Recall@5最高提升16%,达到峰值性能所需微调步数减少40%,冷启动项目召回率提升60%;在需额外CPT的数据集上,性能相当但仅需一半的预训练轮次。结果表明,保留语义几何结构,超越共享前缀,是高效且有效的语义先验。
原文摘要 · Abstract (English)
Recent advances in generative recommendation (GR) leverage large language models (LLMs) as recommender backbones, enabling LLMs to directly generate recommendations conditioned on item-interaction histories. In these systems, items are often represented through semantic IDs (SIDs) added to the LLM vocabulary as special tokens. Ideally, SIDs imbue item token representations with semantic priors, thereby improving model generalization. However, standard vocabulary expansion typically initializes these tokens as random Gaussian vectors, discarding the SIDs' underlying continuous geometry and forcing the LLM to relearn token relationships from interaction data. To demonstrate the consequences of this design, we first show that training from this initialization tends to organize SID embeddings around item popularity rather than semantics. We further show that, despite partially reducing the reliance on popularity and improving cold item performance, the computationally expensive process of continual pretraining (CPT) fails to reliably recover the original semantic geometry. To address these findings, we propose a simple, parameter-free intervention that initializes SID token embeddings directly from their corresponding centroids in the semantic embedding space. Requiring only a few lines of code and no additional training or inference overhead, this drop-in approach improves pure-SFT Recall@5 by up to 16%, reaches peak performance with up to 40% fewer SFT steps, and improves cold-item Recall@5 by up to 60%. Moreover, on datasets that benefit from additional CPT, centroid initialization reaches comparable performance while requiring half as many CPT epochs. Together, our findings show that preserving SID geometry, beyond shared-prefix structure, provides a simple and effective semantic prior for LLM-based GR.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。