用分窗Transformer提升多模态行为序列的垃圾账号检测效率
A Split-Window Transformer for Multi-Model Sequence Spammer Detection using Multi-Model Variational Autoencoder
- 基于多模态VAE的用户行为分词，降低噪声干扰
- 分层分窗注意力机制，将长序列压缩为短窗口计算
- 在公开数据集上超越现有最佳模型，适合作为通用检测框架
本文提出一种新型Transformer——MS²Dformer，可作为多模态序列垃圾账号检测的通用骨干网络。该任务面临双重挑战：一是用户多模态噪声信息干扰特征提取；二是用户历史行为序列过长，导致注意力计算占用大量GPU内存。为此，我们首先设计基于多模态变分自编码器（MVAE）的用户行为分词算法，有效抑制噪声。随后提出分层分窗多头注意力（SW/W-MHA）机制，将超长序列分层转化为窗内短期与窗间全局注意力组合。在公开数据集上预训练后，MS²Dformer性能显著超越此前最优方法，验证了其作为通用骨干网络的有效性。
原文摘要 · Abstract (English)
This paper introduces a new Transformer, called MS$^2$Dformer, that can be used as a generalized backbone for multi-modal sequence spammer detection. Spammer detection is a complex multi-modal task, thus the challenges of applying Transformer are two-fold. Firstly, complex multi-modal noisy information about users can interfere with feature mining. Secondly, the long sequence of users' historical behaviors also puts a huge GPU memory pressure on the attention computation. To solve these problems, we first design a user behavior Tokenization algorithm based on the multi-modal variational autoencoder (MVAE). Subsequently, a hierarchical split-window multi-head attention (SW/W-MHA) mechanism is proposed. The split-window strategy transforms the ultra-long sequences hierarchically into a combination of intra-window short-term and inter-window overall attention. Pre-trained on the public datasets, MS$^2$Dformer's performance far exceeds the previous state of the art. The experiments demonstrate MS$^2$Dformer's ability to act as a backbone.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。