提出NEST模型,更好处理事件序列中的嵌套结构,提升效率与效果。
NEST: Nested Event Stream Transformer for Sequences of Multisets
- 保留事件序列的嵌套结构,避免扁平化带来的计算浪费
- 引入掩码集合建模,增强集合级别表征能力
- 适用于医疗记录等多集合序列数据,适合需要精准表征的场景
事件流数据常呈现层次结构,多个事件同时发生形成多重集序列。例如电子健康记录中,医疗事件按临床就诊分组,具有明确的时间顺序,但单次就诊内的事件顺序和时间可能未知或不可靠。现有基础模型通常将这种层次结构扁平化为一维序列,导致(i)密集注意力带来计算效率低下,并学习到虚假的组内关系;(ii)下游任务依赖启发式后训练池化,生成质量较差的集合级表示。本文表明,保持原始层次结构在模型架构中可提供有益的归纳偏置,从而提升计算效率与表征质量。我们提出嵌套事件流变压器(NEST),一种用于多重集序列事件流的基础模型。在此基础上,构建掩码集合建模(MSM)机制,有效促进集合级表征学习。在真实世界多重集序列数据上的实验表明,NEST不仅捕捉现实动态,还提升了预训练效率与下游性能。
原文摘要 · Abstract (English)
Event stream data often exhibit hierarchical structure in which multiple events co-occur, resulting in a sequence of multisets (i.e., bags of events). In electronic health records (EHRs), for example, medical events are grouped into a sequence of clinical encounters with well-defined temporal structure, but the order and timing of events within each encounter may be unknown or unreliable. Most existing foundation models (FMs) for event stream data flatten this hierarchy into a one-dimensional sequence, leading to (i) computational inefficiency associated with dense attention and learning spurious within-set relationships, and (ii) lower-quality set-level representations from heuristic post-training pooling for downstream tasks. Here, we show that preserving the original hierarchy in the FM architecture provides a useful inductive bias that improves both computational efficiency and representation quality. We then introduce Nested Event Stream Transformer (NEST), a FM for event streams comprised of sequences of multisets. Building on this architecture, we formulate Masked Set Modeling (MSM), an efficient paradigm that promotes improved set-level representation learning. Experiments on real-world multiset sequence data show that NEST captures real-world dynamics while improving both pretraining efficiency and downstream performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。