48B参数模型每轮仅激活2.7B,实现高效推理。
JoyAI-LLM Flash: Advancing Mid-Scale LLMs with Token Efficiency
- 采用MoE架构与分层强化学习,动态调节思考与非思考模式。
- 激活参数仅2.7B,稀疏比高于同类模型,推理吞吐显著提升。
- 支持开源,适合追求高效率的中等规模模型应用者。
我们提出JoyAI-LLM Flash,一种面向50B以下参数规模的高效混合专家(MoE)语言模型,旨在重新定义性能与分词效率之间的平衡。该模型在20万亿令牌的海量语料上预训练,并通过监督微调(SFT)、直接偏好优化(DPO)及跨多环境的大规模强化学习(RL)进行后训练优化。为提升分词效率,模型策略性地平衡了“思考”与“非思考”认知模式,并引入受纤维化理论启发的FiberPO算法,将信任域维护分解为全局与局部组件,实现统一的多尺度稳定性控制。模型总参数量为48B,但每次前向传播仅激活2.7B参数,稀疏比显著优于同规模主流模型。为进一步提升推理吞吐,采用训练-推理联合设计,集成密集多分词预测(MTP)与量化感知训练(QAT)。我们已在Hugging Face发布JoyAI-LLM-48B-A3B Base及其后训练版本的检查点,以支持开源社区。
原文摘要 · Abstract (English)
We introduce JoyAI-LLM Flash, an efficient Mixture-of-Experts (MoE) language model designed to redefine the trade-off between strong performance and token efficiency in the sub-50B parameter regime. JoyAI-LLM Flash is pretrained on a massive corpus of 20 trillion tokens and further optimized through a rigorous post-training pipeline, including supervised fine-tuning (SFT), Direct Preference Optimization (DPO), and large-scale reinforcement learning (RL) across diverse environments. To improve token efficiency, JoyAI-LLM Flash strategically balances \emph{thinking} and \emph{non-thinking} cognitive modes and introduces FiberPO, a novel RL algorithm inspired by fibration theory that decomposes trust-region maintenance into global and local components, providing unified multi-scale stability control for LLM policy optimization. To enhance architectural sparsity, the model comprises 48B total parameters while activating only 2.7B parameters per forward pass, achieving a substantially higher sparsity ratio than contemporary industry leading models of comparable scale. To further improve inference throughput, we adopt a joint training-inference co-design that incorporates dense Multi-Token Prediction (MTP) and Quantization-Aware Training (QAT). We release the checkpoints for both JoyAI-LLM-48B-A3B Base and its post-trained variants on Hugging Face to support the open-source community.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。