用历史令牌提升Transformer对事件序列的分类能力
HT-Transformer: Event Sequences Classification by Accumulating Prefix Information with History Tokens
- 引入历史令牌，逐步累积序列上下文信息
- 在金融、电商、医疗任务中显著超越现有模型
- 适合需要长期依赖建模的序列分类场景
深度学习在建模序列数据方面取得了显著成果，包括事件序列、时间点过程和不规则时间序列。近年来，Transformer已广泛取代循环网络。然而，在预测未来目标的分类任务中，Transformer的表现常不及RNN。本文揭示了这一性能差距的关键原因：Transformer缺乏能紧凑有效表示整个序列的单一状态向量；同时，嵌入向量的对比预训练无法捕捉局部上下文，而这对准确预测至关重要。为此，我们提出历史令牌（history tokens）这一新概念，通过在下一个词预测预训练过程中累积历史信息。该方法显著提升了基于Transformer的模型性能，在金融、电商和医疗任务中表现优异。代码已在GitHub公开。
原文摘要 · Abstract (English)
Deep learning has achieved remarkable success in modeling sequential data, including event sequences, temporal point processes, and irregular time series. Recently, transformers have largely replaced recurrent networks in these tasks. However, transformers often underperform RNNs in classification tasks where the objective is to predict future targets. The reason behind this performance gap remains largely unexplored. In this paper, we identify a key limitation of transformers: the absence of a single state vector that provides a compact and effective representation of the entire sequence. Additionally, we show that contrastive pretraining of embedding vectors fails to capture local context, which is crucial for accurate prediction. To address these challenges, we introduce history tokens, a novel concept that facilitates the accumulation of historical information during next-token prediction pretraining. Our approach significantly improves transformer-based models, achieving impressive results in finance, e-commerce, and healthcare tasks. The code is publicly available on GitHub.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。