ResFormer用分层记忆机制高效处理长文本分类，准确率提升超22%
ResFormer: All-Time Reservoir Memory for Long Sequence Classification
- 融合蓄水池计算与Transformer，分层建模长短依赖
- 在EmoryNLP上比基线模型高22.3%准确率，内存更低
- 适合需要长上下文理解的对话、情感分析任务
序列分类在自然语言处理中至关重要，用于情感分析、意图识别和主题分类等任务。基于Transformer的模型虽表现优异，但因时间与内存复杂度呈二次增长，限制了输入长度。尽管已有大量工作降低计算开销，长上下文处理仍具挑战。为此，我们提出ResFormer，一种新型神经网络架构，通过级联方法高效建模可变长度上下文。ResFormer结合非线性读出的蓄水池计算网络，在线性时间内捕捉长期上下文依赖；同时使用固定输入长度的传统Transformer建模句内短期依赖。实验表明，ResFormer显著优于DeepSeek-Qwen和ModernBERT等基线模型，在EmoryNLP数据集上准确率最高提升22.3%，并在MultiWOZ、MELD和IEMOCAP上保持一致优势。此外，其内存消耗更低，证明了在建模长距离信息方面的高效性与有效性。
原文摘要 · Abstract (English)
Sequence classification is essential in NLP for understanding and categorizing language patterns in tasks like sentiment analysis, intent detection, and topic classification. Transformer-based models, despite achieving state-of-the-art performance, have inherent limitations due to quadratic time and memory complexity, restricting their input length. Although extensive efforts have aimed at reducing computational demands, processing extensive contexts remains challenging. To overcome these limitations, we propose ResFormer, a novel neural network architecture designed to model varying context lengths efficiently through a cascaded methodology. ResFormer integrates an reservoir computing network featuring a nonlinear readout to effectively capture long-term contextual dependencies in linear time. Concurrently, short-term dependencies within sentences are modeled using a conventional Transformer architecture with fixed-length inputs. Experiments demonstrate that ResFormer significantly outperforms baseline models of DeepSeek-Qwen and ModernBERT, delivering an accuracy improvement of up to +22.3% on the EmoryNLP dataset and consistent gains on MultiWOZ, MELD, and IEMOCAP. In addition, ResFormer exhibits reduced memory consumption, underscoring its effectiveness and efficiency in modeling extensive contextual information.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。