26亿参数模型通过新架构提升长文本理解与生成准确性
Motif 2.6B Technical Report
- 采用差分注意力与PolyNorm激活函数优化模型结构
- 在多个基准上表现优于同类规模先进模型
- 适合资源有限团队快速部署高效大模型
近期大型语言模型（LLMs）的发展推动了人工智能的革新，但开发兼具高性能与计算效率的基础模型对新兴研究团队仍具挑战。为填补这一空白，我们提出Motif-2.6B，一个26亿参数的基础模型，旨在实现先进LLM能力的普惠化。该模型引入差分注意力和PolyNorm激活函数等创新架构改进，提升了长上下文理解能力，减少幻觉现象，并增强上下文学习性能。通过大量实验验证多种新型组件，最终确定最优架构。全面评估表明，Motif-2.6B在多个基准测试中持续达到或超越同规模顶尖模型的表现，展现出卓越的有效性、可扩展性与实际应用价值。通过详尽实验与定制技术，Motif-2.6B显著推进了高效、可扩展且强大的基础语言模型发展，为未来研究与部署提供了重要参考。
原文摘要 · Abstract (English)
Recent advancements in Large Language Models (LLMs) have revolutionized artificial intelligence, yet developing an effective foundational LLM that balances high performance with computational efficiency remains challenging, especially for emerging research groups. To address this gap, we introduce Motif-2.6B, a 2.6-billion-parameter foundation model designed to democratize advanced LLM capabilities. Motif-2.6B incorporates several innovative architectural enhancements, including Differential Attention and PolyNorm activation functions, which improve long-context comprehension, reduce hallucination, and enhance in-context learning capabilities. We rigorously tested multiple novel architectural components through extensive experimentation to determine the optimal architecture for Motif-2.6B. Comprehensive evaluations demonstrate that Motif-2.6B consistently meets or exceeds the performance of similarly sized state-of-the-art models across diverse benchmarks, showcasing its effectiveness, scalability, and real-world applicability. Through detailed experiments and tailored techniques, Motif-2.6B significantly advances the landscape of efficient, scalable, and powerful foundational LLMs, offering valuable insights and a robust foundation for future research and deployment.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。