通过分层注意力掩码，让大模型更好捕捉用户行为的协同信号。
HatLLM: Hierarchical Attention Masking for Enhanced Collaborative Modeling in LLM-based Recommendation
- 浅层屏蔽跨物品注意力，强化物品内部语义理解
- 深层屏蔽物品内注意力，促进跨物品关联建模
- 在三个真实数据集上平均提升9.13%，适合推荐系统研究者
近年来，利用大语言模型（LLMs）进行序列推荐的研究迅速发展。LLMs在通过细粒度语义推理推断用户细微偏好方面展现出巨大潜力，但其在有效建模协同信号——即用户历史交互中固有的行为相关性——方面存在明显不足。我们的实证分析进一步发现，LLMs中的注意力机制倾向于过度关注同一物品内的标记，从而阻碍了跨物品相关性的捕捉。为此，我们提出一种新的分层注意力掩码策略，称为HatLLM。具体而言，在浅层中，HatLLM屏蔽不同物品间标记的注意力，以促进物品内部的语义理解；而在深层中，它则屏蔽物品内部的注意力，迫使模型捕捉跨物品的相关性。这种逐层推进的机制使LLMs能够联合建模标记级和物品级依赖关系。在三个真实世界数据集上的大量实验表明，与现有基于LLM的方法相比，HatLLM平均性能提升9.13%。
原文摘要 · Abstract (English)
Recent years have witnessed a surge of research on leveraging large language models (LLMs) for sequential recommendation. LLMs have demonstrated remarkable potential in inferring users' nuanced preferences through fine-grained semantic reasoning. However, they also exhibit a notable limitation in effectively modeling collaborative signals, i.e., behavioral correlations inherent in users' historical interactions. Our empirical analysis further reveals that the attention mechanisms in LLMs tend to disproportionately focus on tokens within the same item, thereby impeding the capture of cross-item correlations. To address this limitation, we propose a novel hierarchical attention masking strategy for LLM-based recommendation, termed HatLLM. Specifically, in shallow layers, HatLLM masks attention between tokens from different items, facilitating intra-item semantic understanding; in contrast, in deep layers, HatLLM masks attention within items, thereby compelling the model to capture cross-item correlations. This progressive, layer-wise approach enables LLMs to jointly model both token-level and item-level dependencies. Extensive experiments on three real-world datasets demonstrate that HatLLM achieves significant performance gains (9.13% on average) over existing LLM-based methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。