让大模型记住超长文本，关键在高效注意力机制。
Every Token Counts: Generalizing 16M Ultra-Long Context in Large Language Models
- 采用分层稀疏注意力，实现高效长文本处理。
- 在1600万词上下文任务中，检索准确率超90%。
- 适合需要超长记忆的智能系统研究与应用。
本文探讨构建「能记忆的机器」的挑战，将长期记忆问题归结为高效的超长上下文建模。我们提出三个关键属性：稀疏性、随机访问灵活性和长度泛化能力。为此，提出分层稀疏注意力（HSA）机制，满足上述所有特性。将HSA融入Transformer，构建80亿参数的MoE模型HSA-UltraLong，该模型在超过8万亿词的语料上训练，并在不同任务中评估其对域内与域外上下文长度的处理能力。结果表明，该模型在域内长度上表现接近全注意力基线，在1600万词上下文的多数上下文检索任务中准确率超过90%。本报告总结实验洞察与开放问题，为未来超长上下文建模研究奠定基础。
原文摘要 · Abstract (English)
This work explores the challenge of building ``Machines that Can Remember'', framing long-term memory as the problem of efficient ultra-long context modeling. We argue that this requires three key properties: \textbf{sparsity}, \textbf{random-access flexibility}, and \textbf{length generalization}. To address ultra-long-context modeling, we leverage Hierarchical Sparse Attention (HSA), a novel attention mechanism that satisfies all three properties. We integrate HSA into Transformers to build HSA-UltraLong, which is an 8B-parameter MoE model trained on over 8 trillion tokens and is rigorously evaluated on different tasks with in-domain and out-of-domain context lengths to demonstrate its capability in handling ultra-long contexts. Results show that our model performs comparably to full-attention baselines on in-domain lengths while achieving over 90\% accuracy on most in-context retrieval tasks with contexts up to 16M. This report outlines our experimental insights and open problems, contributing a foundation for future research in ultra-long context modeling.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。