提出LPC-SM模型，用分离机制提升长文本建模效果。
LPC-SM: Local Predictive Coding and Sparse Memory for Long-Context Language Modeling
- 将局部注意力、持久记忆等模块分置于同一结构块中
- 在4096词长下保持稳定，损失降至11.582
- 适合研究长序列建模与注意力替代方案的学者
当前多数长上下文语言模型仍依赖注意力机制处理局部交互与远距离状态，限制了序列建模新范式的探索。本文提出LPC-SM，一种混合自回归架构，将局部注意力、持久记忆、预测修正和运行时控制在同一模块内分离实现，并采用正交新颖性传输（ONT）管理慢速记忆写入。在三个阶段评估：基础语言建模、数学续写及4096词续写。移除mHC使阶段A最终语言模型损失从12.630升至15.127；自适应稀疏控制相较固定比例续写，将阶段B损失从12.137降至10.787。完整流程在4096序列长度下保持稳定，阶段C最终损失为11.582，延迟标识诊断交叉熵由14.396降至12.031。结果表明，长上下文自回归建模可超越注意力单一分工。
原文摘要 · Abstract (English)
Most current long-context language models still rely on attention to handle both local interaction and long-range state, which leaves relatively little room to test alternative decompositions of sequence modeling. We propose LPC-SM, a hybrid autoregressive architecture that separates local attention, persistent memory, predictive correction, and run-time control within the same block, and we use Orthogonal Novelty Transport (ONT) to govern slow-memory writes. We evaluate a 158M-parameter model in three stages spanning base language modeling, mathematical continuation, and 4096-token continuation. Removing mHC raises the Stage-A final LM loss from 12.630 to 15.127, while adaptive sparse control improves the Stage-B final LM loss from 12.137 to 10.787 relative to a matched fixed-ratio continuation. The full route remains stable at sequence length 4096, where Stage C ends with final LM loss 11.582 and improves the delayed-identifier diagnostic from 14.396 to 12.031 in key cross-entropy. Taken together, these results show that long-context autoregressive modeling can be organized around a broader division of labor than attention alone.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。