用时间核提升长序列记忆，让模型高效存取连续信息。
Long-Sequence Memory with Temporal Kernels and Dense Hopfield Functionals
- 引入时间核函数建模序列依赖，增强长序列存储能力。
- 在电影帧数据上实现高维序列的精准顺序检索。
- 适合需要长期依赖建模的NLP与时间序列任务。
本文提出一种新型能量函数用于长序列记忆，基于密集霍普菲尔德网络框架，通过高阶交互实现指数级存储容量。在此基础上，我们设计了时间核 $K(m, k)$ 以捕捉时间依赖性，实现对长序列模式的高效顺序检索。实验表明，该方法可成功存储并按序恢复电影帧，得益于每帧高维向量带来的显著空间差异。该技术适用于现代Transformer架构，可实现高效长序列建模、记忆增强、带时间偏置的注意力优化，以及时间序列中长期依赖的更好处理。本模型为解决Transformer在长上下文任务中的局限提供了新路径，对自然语言处理、预测等任务具有潜在应用价值。
原文摘要 · Abstract (English)
In this study we introduce a novel energy functional for long-sequence memory, building upon the framework of dense Hopfield networks which achieves exponential storage capacity through higher-order interactions. Building upon earlier work on long-sequence Hopfield memory models, we propose a temporal kernal $K(m, k)$ to incorporate temporal dependencies, enabling efficient sequential retrieval of patterns over extended sequences. We demonstrate the successful application of this technique for the storage and sequential retrieval of movies frames which are well suited for this because of the high dimensional vectors that make up each frame creating enough variation between even sequential frames in the high dimensional space. The technique has applications in modern transformer architectures, including efficient long-sequence modeling, memory augmentation, improved attention with temporal bias, and enhanced handling of long-term dependencies in time-series data. Our model offers a promising approach to address the limitations of transformers in long-context tasks, with potential implications for natural language processing, forecasting, and beyond.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。