用自监督预训练提升异构事件序列的预测能力，显著优于现有方法。
SOHET: Sequence Of Heterogeneous Events Transformer with Self-Supervised Pre-Training
- 分类型表格编码+时序嵌入，结合因果或双向Transformer处理异构事件流。
- 在17类事件的欺诈检测任务中，性能比基线高5.8%，预训练再提2.6%。
- 适用于大规模实时事件分析，尤其适合金融风控与推荐系统场景。
许多机器学习应用依赖异构事件流进行预测，或按事件到达顺序（因果）建模，或对完整序列（双向）建模。本文提出SOHET（异构事件序列变压器），一种分层架构，将事件类型特定的表格编码器与时间及类型嵌入结合，由因果或双向Transformer处理。针对因果场景，设计三种自监督预训练目标。在包含17种事件类型的私有大规模真实世界预订平台欺诈检测任务中，SOHET相较FlexTPP、NAPPT和CIPPT提升5.8%。预训练带来额外2.6%性能增益，并实现2.4%更快收敛。在EBES基准上，双向SOHET在8个任务中有6个达到或超过已有最佳表现。
原文摘要 · Abstract (English)
Many machine learning applications rely on heterogeneous event streams to make predictions, either causally as events arrive or bidirectionally over complete sequences. We propose SOHET (Sequence Of Heterogeneous Events Transformer), a hierarchical architecture combining event-type-specific tabular encoders with temporal and type embeddings, processed by a causal or bidirectional transformer. We introduce three self-supervised pre-training objectives for the causal setting. On a proprietary large-scale real-world Booking.com fraud detection task with 17 event types, SOHET outperforms FlexTPP, NAPPT, and CIPPT by 5.8%. Pre-training yields an additional 2.6% gain and 2.4% faster convergence. On the EBES benchmark, bidirectional SOHET matches or exceeds the published best on 6 out of 8 tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。