无需人工选行为词表,自动预训练用户行为序列
Bootstrapping your behavior: a new pretraining strategy for user behavior sequence data
- 用自动生成的嵌入向量替代人工构建词表,实现端到端预训练
- 在8个下游任务中平均提升3.9% AUC,训练速度提高近100倍
- 无监督下生成有意义注意力模式,适合工业级用户行为建模
用户行为序列(UBS)建模在工业应用中至关重要。随着数据规模和任务多样性增长,UBS预训练方法日益关键。现有方法依赖预测行为分布,核心步骤是构建选定的行为词表,但该过程需人工干预且易引入偏差,词表容量限制也影响模型泛化能力。本文提出新型预训练策略Bootstrapping Your Behavior( model{}),通过预测未来时间窗口内所有行为信息的自动构建监督嵌入,避免了人工词表选择。实现上采用学生-教师编码器结构,有效生成预训练监督信号。在两个真实工业数据集和八个下游任务上的实验表明, model{}平均提升3.9% AUC,训练吞吐量提高98.9%。值得注意的是,模型在无标签监督下展现出有意义的注意力模式与聚类表示。在线部署两个月期间,该模型在支付宝两个金融逾期风险预测任务中分别将KS提升2.7%和7.1%,为蚂蚁集团减少数百万美元坏账风险。
原文摘要 · Abstract (English)
User Behavior Sequence (UBS) modeling is crucial in industrial applications. As data scale and task diversity grow, UBS pretraining methods have become increasingly pivotal. State-of-the-art UBS pretraining methods rely on predicting behavior distributions. The key step in these methods is constructing a selected behavior vocabulary. However, this manual step is labor-intensive and prone to bias. The limitation of vocabulary capacity also directly affects models' generalization ability. In this paper, we introduce Bootstrapping Your Behavior (\model{}), a novel UBS pretraining strategy that predicts an automatically constructed supervision embedding summarizing all behaviors' information within a future time window, eliminating the manual behavior vocabulary selection. In implementation, we incorporate a student-teacher encoder scheme to construct the pretraining supervision effectively. Experiments on two real-world industrial datasets and eight downstream tasks demonstrate that \model{} achieves an average improvement of 3.9\% in AUC and 98.9\% in training throughput. Notably, the model exhibits meaningful attention patterns and cluster representations during pretraining without any label supervision. In our online deployment over two months, the pretrained model improves the KS by about 2.7\% and 7.1\% over the baseline model for two financial overdue risk prediction tasks in the Alipay mobile application, which reduces bad debt risk by millions of dollars for Ant group.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。