2500亿参数大模型,支持百万词长上下文,专攻复杂智能体任务。
Solar Open 2 Technical Report

- 混合专家架构结合新型注意力堆叠,实现无位置编码的百万词上下文。
- 用10万亿高质量数据训练,在同等算力下性能超越前代模型。
- 擅长多领域智能体任务,韩语评测表现媲美大型闭源模型。
我们提出 Solar Open 2,一个 2500 亿参数(250B-A15B)的混合专家语言模型,专为长时序智能体任务设计,由 Solar Open 1(100B)扩展而来。为在单个上下文中容纳完整智能体轨迹,模型采用混合注意力结构:每三个线性注意力层插入一层 softmax 层,不使用位置编码,并将门控增量规则扩展至负特征值,实现 100 万词(1M-token)上下文窗口。在固定算力预算下提升训练效率:一是从 Solar Open 1 初始化,复用 56.9 亿参数共享骨架;二是通过质量与稀有度感知的数据筛选及混合比例优化,将 20 万亿(20T)数据集精炼为 10 万亿(10T)高价值混合数据。通过在定制场景中训练十二个领域专家,再以多教师在线蒸馏(MOPD)整合为单一模型。在英文基准测试中,优于同规模开源模型,在 MMLU-Pro、LiveCodeBench 及 APEX-Agents 智能体套件上领先;在韩语评测中,平均得分最高,于 Ko-GDPval(自研韩语办公代理基准)上表现接近 1.6 万亿参数的 DeepSeek-V4-Pro,仅为其六分之一大小。
原文摘要 · Abstract (English)
We present Solar Open 2, a 250B-A15B Mixture-of-Experts language model built for long-horizon agentic tasks, scaled up from Solar Open 1 (Solar Open 100B). To hold entire agent trajectories in a single context, Solar Open 2 reaches a 1M-token window through a hybrid attention stack that interleaves one softmax layer among every three linear-attention layers, using no positional encoding and a gated delta rule extended to negative eigenvalues. To train at this scale under a fixed compute budget, we make training efficient in two ways: a stronger starting point, and higher-value data. For the starting point, we initialize Solar Open 2 from Solar Open 1, transferring the 5.69B-parameter shared skeleton that survives the architectural change and learning everything else through full pre-training. For the data, we curate for value per token: quality- and rarity-aware data curation and mixture-ratio optimization refine a 20T pool into a 10T mixture that, at equal token budget, outperforms the Solar Open 1 recipe. To build its agent skills, we train twelve domain specialists across purpose-built scenarios, then consolidate them into a single model by Multi-teacher On-Policy Distillation (MOPD). Against comparably sized open-weight models on English benchmarks, Solar Open 2 leads on MMLU-Pro, LiveCodeBench, and the APEX-Agents agentic suite, and stays competitive with the strongest (DeepSeek-V4-Flash and MiMo-V2.5) elsewhere. On Korean benchmarks, Solar Open 2 records the highest average of any model compared, including fast-tier closed APIs, and on Ko-GDPval, an in-house Korean officework-agent benchmark, it is competitive with DeepSeek-V4-Pro (1.6T) at less than a sixth of its size.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。