发现序列拆分会误导推荐模型评估结果,需谨慎使用。
Pay Attention to Sequence Split: Uncovering the Impacts of Sub-Sequence Splitting on Sequential Recommendation Models
- 通过分析不同拆分方法与损失函数的组合效果,揭示其对模型性能的关键影响。
- 不当组合可能导致性能下降,甚至低于经典模型。
- 适合关注推荐系统评估公平性与数据预处理细节的研究者。
子序列拆分(SSS)被证明可有效缓解序列推荐(SR)中的数据稀疏问题,通过将原始用户交互序列拆分为多个子序列提升模型表现。然而本文发现:(i) SSS可能干扰模型真实性能的评估,许多近期先进模型在数据读取阶段隐式使用了SSS,移除后性能显著下降,甚至低于早期经典模型;(ii) SSS的有效性高度依赖于特定拆分方法、目标策略与损失函数的协同作用,不恰当组合反而损害性能;(iii) SSS之所以表现优异,是因为它均衡了训练数据分布并提高了不同物品被预测的可能性。文章提出应对策略,讨论数据增强方法及未来方向,呼吁社区重新审视数据拆分对推荐评估的影响。代码与数据将在论文接受后公开。
原文摘要 · Abstract (English)
Sub-sequence splitting (SSS) has been demonstrated as an effective approach to mitigate data sparsity in sequential recommendation (SR) by splitting a raw user interaction sequence into multiple sub-sequences. Previous studies have demonstrated its ability to enhance the performance of SR models significantly. However, in this work, we discover that \textbf{(i). SSS may interfere with the evaluation of the model's actual performance.} We observed that many recent state-of-the-art SR models employ SSS during the data reading stage (not mentioned in the papers). When we removed this operation, performance significantly declined, even falling below that of earlier classical SR models. The varying improvements achieved by SSS and different splitting methods across different models prompt us to analyze further when SSS proves effective. We find that \textbf{(ii). SSS demonstrates strong capabilities only when specific splitting methods, target strategies, and loss functions are used together.} Inappropriate combinations may even harm performance. Furthermore, we analyze why sub-sequence splitting yields such remarkable performance gains and find that \textbf{(iii). it evens out the distribution of training data while increasing the likelihood that different items are targeted.} Finally, we provide suggestions for overcoming SSS interference, along with a discussion on data augmentation methods and future directions. We hope this work will prompt the broader community to re-examine the impact of data splitting on SR and promote fairer, more rigorous model evaluation. All analysis code and data will be made available upon acceptance. We provide a simple, anonymous implementation at https://github.com/KingGugu/SSS4SR.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。