arXiv:2608.30181cs.AIcs.CL2026-08

6880亿参数大模型,高效支持长文本和智能应用。

A.X K2 Technical Report

论文配图:A.X K2 Technical Report
图 1 · 摘自论文原文
  • 用稀疏门控注意力与门控归一化,提升训练稳定性和长文本效率。
  • 在256K上下文下RULER评测达94.6分,4比特推理误差小于1点。
  • 支持思维模式切换,适合需要智能决策的复杂任务场景。

我们提出A.X K2,一个6880亿参数的专家混合(MoE)语言模型,专为智能体应用设计。该模型在约8.5万亿词上从零训练,数据量少于前代A.X K1,但质量更高,且大幅扩充了智能体与软件工程数据,性能全面超越前代,部分基准提升超30个百分点,体现显著的令牌效率优势。为高效支持长上下文,引入稀疏门控注意力(SGA),结合稀疏注意力与门控注意力,并采用门控归一化(GN)稳定大规模训练。SGA通过稀疏索引器预热,在128K上下文原生训练,每个查询仅读取2048个位置,但长文本表现不变,A.X K2在RULER测试中256K上下文得分达94.6。GN有效抑制异常值,使4比特NVFP4推理精度仅比FP8低1点。简单的思维融合策略允许用户在单一模型中自由切换思考与非思考模式。大量评估表明,A.X K2在数学与韩语基准上表现优于或匹配强开源基线。

原文摘要 · Abstract (English)

We introduce A.X K2, a 688B-parameter Mixture-of-Experts (MoE) language model trained from scratch as a high-performance foundation for \emph{agentic} applications. Trained on approximately 8.5T tokens---fewer than its predecessor, A.X K1---on a smaller but higher-quality mixture with substantially expanded agentic and software-engineering data, it nonetheless improves over A.X K1 across the board, by over 30 percentage points on some benchmarks, reflecting large gains in token efficiency. To support long contexts efficiently, we introduce Sparse Gated Attention (SGA), which combines sparse attention with gated attention, and adopt Gated Norm (GN) to stabilize large-scale training. SGA is trained natively at 128K through a \emph{sparse} indexer warmup that optimizes the indexer against its own sparse top-$k$ selection rather than the dense attention distribution, making adaptation markedly cheaper: each query reads only 2,048 positions, yet long-context quality is unchanged and A.X K2 scores 94.6 on RULER out to 256K. The outlier suppression of GN in turn keeps 4-bit NVFP4 serving within one point of FP8 accuracy. A simple yet effective Think-Fusion recipe further lets users switch between thinking and non-thinking modes within a single unified model. Extensive evaluations show that A.X K2 performs competitively against strong open-weight baselines, matching or exceeding them on math and Korean-language benchmarks.

大模型长文本智能体高效推理

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。