让自回归状态空间模型同时学事件和时间,提升时序推理能力。
ChronoSSM: Training for Temporally Aware Representations in Autoregressive State Space Models
- 联合训练事件与时间信号,共享模型主干
- 冻结表示后仍能更好恢复事件间隔信息
- 适合需要精准时序建模的任务场景
现代序列模型(如Transformer、状态空间模型)虽在生成建模中表现强大,但通常只关注事件本身,将时间视为次要信息。在包含明确时间戳的数据挖掘场景中,这种分离会限制时序推理、异常检测与事件顺序重建。现有方法常采用两阶段策略:先用事件预测学习表示,再单独训练时间模型。但该方法假设事件表示已蕴含足够时序结构,可能不成立。本文提出ChronoSSM,一种自回归状态空间模型(SSM),通过联合优化事件与时间生成目标,使模型主干同时学习事件与时间信息。对比实验显示,在四个不同领域(覆盖密集与部分时间监督)中,联合训练相比两阶段方法,能在不降低内容生成质量的前提下,显著提升冻结表示中对事件间隔的可恢复性。结果表明,时间监督可增强表示的时序信息,且不影响事件建模效果。
原文摘要 · Abstract (English)
Modern sequence models, from Transformers to State Space Models, have enabled powerful generative modeling across diverse domains, yet they are typically trained to predict what happens while treating when it happens as a secondary concern. In data-mining settings where events are associated with explicit timing information, this separation can limit temporal reasoning, anomaly detection, and faithful reconstruction of event chronology. A common strategy is to treat timing as an auxiliary signal, training a separate timing model using representations learned solely for event prediction. However, this two-stage approach implicitly assumes that representations optimized for event prediction already contain sufficient temporal structure. We introduce ChronoSSM, an autoregressive State Space Model (SSM) that jointly models events and timestamps with a shared backbone trained using combined token and temporal generation objectives. We compare the joint regime, where temporal supervision updates the backbone, with the two-stage regime, where timing is learned only using the frozen event representations. Across four domains spanning dense and partial timestamp supervision, joint training consistently makes inter-arrival information more recoverable from frozen representations without any systematic degradation in content-generation quality overall. Our results show that temporal supervision can produce more temporally informative representations without materially degrading autoregressive event modeling.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。