70亿参数模型实现512K上下文，高效处理长文本任务
Scaling Context, Not Parameters: Training a Compact 7B Language Model for Efficient Long-Context Processing
- 通过优化训练策略，让小模型支持超长上下文
- 在3个基准上表现优异，尤其在长程推理任务中领先
- 开源且下载量破十万，适合实际长文本应用
我们提出MegaBeam-Mistral-7B，一个支持512K token上下文长度的70亿参数语言模型。该工作解决了长上下文训练中的实际限制，适用于合规监控、验证等真实任务。在三个长上下文基准测试中，该模型在HELMET上展现出优越的上下文学习能力，在RULER上具备强检索与追踪性能。目前是唯一无需RAG或微调即可在512K上下文长度下实现有竞争力长程推理的开源模型。模型已以Apache 2.0许可证完全开源，上线后在Hugging Face下载量超过10万次。
原文摘要 · Abstract (English)
We present MegaBeam-Mistral-7B, a language model that supports 512K-token context length. Our work addresses practical limitations in long-context training, supporting real-world tasks such as compliance monitoring and verification. Evaluated on three long-context benchmarks, our 7B-parameter model demonstrates superior in-context learning performance on HELMET and robust retrieval and tracing capability on RULER. It is currently the only open model to achieve competitive long-range reasoning on BABILong at 512K context length without RAG or targeted fine-tuning. Released as fully open source under the Apache 2.0 license, the model has been downloaded over 100,000 times on Hugging Face. Model available at: https://huggingface.co/aws-prototyping/MegaBeam-Mistral-7B-512k
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。