通过信息增益量化令牌重要性,提升大模型推荐精度。
IGD: Token Decisiveness Modeling via Information Gain in LLMs for Personalized Recommendation
- 用信息增益衡量每个令牌对推荐结果的决策贡献度
- 低信息增益令牌在训练和解码中被降权,高贡献令牌优先保留
- 在4个数据集上显著提升推荐排序指标,适用于大模型推荐系统
大型语言模型(LLM)通过将物品预测视为逐标记的语言生成任务,在推荐领域展现出巨大潜力。然而,现有方法对所有物品标记一视同仁,仅追求似然最大化,忽略了标记层面的决策重要性差异——许多标记对物品区分作用微弱,却在优化和解码中占据主导地位。为此,本文提出一种新视角:将物品生成建模为决策过程,利用信息增益(IG)量化每个标记减少生成物品不确定性所贡献的决策力。实证分析发现,多数标记具有低IG值,但对应高逻辑值(logits),过度影响训练损失与解码过程,可能损害模型性能。基于此,我们提出基于信息增益的决策感知令牌处理策略(IGD),将令牌决策力融入调优与解码阶段:IGD在调优时降低低IG标记权重,在解码时重新平衡以突出高IG标记。该方法超越单纯似然最大化,有效聚焦高决策力标记。在四个基准数据集及两种LLM骨干网络上的大量实验表明,IGD持续提升推荐准确率,在广泛使用的排名指标上显著优于强基线。
原文摘要 · Abstract (English)
Large Language Models (LLMs) have shown strong potential for recommendation by framing item prediction as a token-by-token language generation task. However, existing methods treat all item tokens equally, simply pursuing likelihood maximization during both optimization and decoding. This overlooks crucial token-level differences in decisiveness-many tokens contribute little to item discrimination yet can dominate optimization or decoding. To quantify token decisiveness, we propose a novel perspective that models item generation as a decision process, measuring token decisiveness by the Information Gain (IG) each token provides in reducing uncertainty about the generated item. Our empirical analysis reveals that most tokens have low IG but often correspond to high logits, disproportionately influencing training loss and decoding, which may impair model performance. Building on these insights, we introduce an Information Gain-based Decisiveness-aware Token handling (IGD) strategy that integrates token decisiveness into both tuning and decoding. Specifically, IGD downweights low-IG tokens during tuning and rebalances decoding to emphasize tokens with high IG. In this way, IGD moves beyond pure likelihood maximization, effectively prioritizing high-decisiveness tokens. Extensive experiments on four benchmark datasets with two LLM backbones demonstrate that IGD consistently improves recommendation accuracy, achieving significant gains on widely used ranking metrics compared to strong baselines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。