arXiv:2503.01001cs.IR2025-03

提出新训练方法,让大模型预测点击率快92%且不丢精度。

Towards An Efficient LLM Training Paradigm for CTR Prediction

  • 用动态目标隔离法并行处理多个预测目标,降低计算开销。
  • 在三个公开数据集上训练时间从70.5小时降至5.31小时,平均提速92%。
  • 适合需要高效训练大模型的推荐系统研究者和工业应用团队。

大型语言模型(LLMs)在基于排序的推荐系统中展现出巨大潜力,多项研究证明其在点击率(CTR)预测任务上显著优于传统方法。然而,现有训练范式存在计算效率低下问题,尤其在长用户行为序列下难以应用。当前主流的滑动窗口策略对每条用户行为生成独立训练样本,导致整体复杂度达 $O(mn^2)$,随交互长度线性增长,使训练成本急剧上升。为此,本文提出动态目标隔离(DTI)训练范式,可结构化并行处理 $k$($k >> 1$)个目标行为。同时识别出隐藏状态泄露与位置偏差过拟合两大瓶颈,提出轻量级解决方案有效缓解。在三个主流公开CTR数据集上的实验表明,DTI将训练时间平均减少92%(如从70.5小时降至5.31小时),且不影响预测性能。

原文摘要 · Abstract (English)

Large Language Models (LLMs) have demonstrated tremendous potential as the next-generation ranking-based recommendation system. Many recent works have shown that LLMs can significantly outperform conventional click-through-rate (CTR) prediction approaches. Despite such promising results, the computational inefficiency inherent in the current training paradigm makes it particularly challenging to train LLMs for ranking-based recommendation tasks on large datasets. To train LLMs for CTR prediction, most existing studies adopt the prevalent ''sliding-window'' paradigm. Given a sequence of $m$ user interactions, a unique training prompt is constructed for each interaction by designating it as the prediction target along with its preceding $n$ interactions serving as context. In turn, the sliding-window paradigm results in an overall complexity of $O(mn^2)$ that scales linearly with the length of user interactions. Consequently, a direct adoption to train LLMs with such strategy can result in prohibitively high training costs as the length of interactions grows. To alleviate the computational inefficiency, we propose a novel training paradigm, namely Dynamic Target Isolation (DTI), that structurally parallelizes the training of $k$ (where $k >> 1$) target interactions. Furthermore, we identify two major bottlenecks - hidden-state leakage and positional bias overfitting - that limit DTI to only scale up to a small value of $k$ (e.g., 5) then propose a computationally light solution to effectively tackle each. Through extensive experiments on three widely adopted public CTR datasets, we empirically show that DTI reduces training time by an average of $\textbf{92%}$ (e.g., from $70.5$ hrs to $5.31$ hrs), without compromising CTR prediction performance.

大模型推荐系统高效训练CTR预测

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。