Instella是全开源三亿参数大模型，性能媲美顶尖闭源模型。
Instella: Fully Open Language Models with Stellar Performance
- 全开源训练数据与代码，使用AMD GPU大规模预训练
- 参数量小但性能达全开源模型顶尖，128K长文本支持
- 适合追求透明、可复现研究的开发者和学术界
大型语言模型在多种任务中表现出色，但多数高性能模型仍为闭源或部分开放，限制了透明性与可复现性。本文提出Instella，一个完全开源的三亿参数语言模型，其训练数据与代码库均为公开。基于AMD Instinct MI300X GPU，通过大规模预训练、通用指令微调及人类偏好对齐实现优化。尽管预训练令牌数远少于同类模型，Instella在全开源模型中达到领先性能，且与同规模开权重模型相当。我们还发布两个专用版本：Instella-Long支持长达128K令牌的上下文处理，Instella-Math通过监督微调与强化学习提升数学推理能力。这些成果为社区提供了透明、高效且多功能的语言建模替代方案，推动开放可复现研究的发展。
原文摘要 · Abstract (English)
Large language models (LLMs) have demonstrated remarkable performance across a wide range of tasks, yet the majority of high-performing models remain closed-source or partially open, limiting transparency and reproducibility. In this work, we introduce Instella, a family of fully open three billion parameter language models trained entirely on openly available data and codebase. Powered by AMD Instinct MI300X GPUs, Instella is developed through large-scale pre-training, general-purpose instruction tuning, and alignment with human preferences. Despite using substantially fewer pre-training tokens than many contemporaries, Instella achieves state-of-the-art results among fully open models and is competitive with leading open-weight models of comparable size. We further release two specialized variants: Instella-Long, capable of handling context lengths up to 128K tokens, and Instella-Math, a reasoning-focused model enhanced through supervised fine-tuning and reinforcement learning on mathematical tasks. Together, these contributions establish Instella as a transparent, performant, and versatile alternative for the community, advancing the goal of open and reproducible language modeling research.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。