4200万参数西班牙语网络安全大模型，支持原生工具调用
VectraYX-Nano: A 42M-Parameter Spanish Cybersecurity Language Model with Curriculum Learning and Native Tool Use
- 从零训练的4200万参数模型，分三阶段课程学习，逐步提升性能
- 在西班牙语安全数据集上达到对话能力0.725，工具调用能力0.230
- 首个支持原生工具调用的西班牙语网络安全模型，适合本地部署
我们提出VectraYX-Nano，一个41.95M参数的解码器架构西班牙语网络安全语言模型，聚焦拉丁美洲地区，并通过模型上下文协议（MCP）实现原生工具调用。该模型有四项贡献：(i) 构建了170M token的西班牙语语料库VectraYX-Sec-ES，使用8台虚拟机分布式管道生成，耗资约25美元云算力，分为三个课程阶段（对话42M、网络安全118M、攻击工具10M）；(ii) 采用42M参数的Transformer解码器，包含GQA、QK-Norm、RMSNorm、SwiGLU、RoPE和z-loss，搭配16,384词元的字节回退BPE；(iii) 三阶段课程学习并引入重播机制，损失单调下降（9.80 → 3.17 → 3.00 → 2.16）；微调后损失降至1.74，基准测试在B5上达0.775±0.043（N=4种子），阶段2重播比例≥25%可饱和提升；(iv) 实验发现：低困惑度预训练语料反而导致对话能力下降（v2>v4>v6）；工具选择基础为0.000是语料密度问题，非模型容量限制——调整微调混合比例至1:21后，模型达B4=0.230±0.052，同时保持B1=0.332±0.005和B5=0.725±0.130；LoRA微调260M模型可达0.445±0.201。发布的GGUF模型为96MB F16格式，在普通硬件下推理延迟<1秒，据我们所知是首个端到端集成MCP的西班牙语原生网络安全大模型。
原文摘要 · Abstract (English)
We present VectraYX-Nano, a 41.95M-parameter decoder-only language model trained from scratch in Spanish for cybersecurity, with a Latin-American regional focus and native tool invocation via the Model Context Protocol (MCP). The model has four contributions. (i) Corpus: VectraYX-Sec-ES, a 170M-token Spanish corpus assembled by an eight-VM distributed pipeline at ~$25 USD of cloud compute and split into three curriculum phases (conversational 42M, cybersecurity 118M, offensive tooling 10M). (ii) Architecture: a 42M Transformer decoder with GQA, QK-Norm, RMSNorm, SwiGLU, RoPE and z-loss, paired with a domain-balanced 16,384-token byte-fallback BPE. (iii) Curriculum with replay across the three phases yields a monotonic loss descent (9.80 -> 3.17 -> 3.00 -> 2.16); after SFT (loss 1.74) the v2 bootstrap-ablation reference attains a conversational gate of 0.775 +/- 0.043 on B5 over N=4 seeds, and a controlled Phase-2 replay sweep over {0,5,10,25,50}% saturates B5 at >=25% replay. (iv) Two empirical findings, both N=4. A controlled bootstrap-corpus ablation across v2 (OpenSubs), v4 (mC4-ES), and v6 (60/25/15 OpenSubs/mC4/Wiki) exposes a loss-versus-register inversion: lower-perplexity bootstraps yield measurably worse conversational behavior (v2 > v4 > v6 on B5 at every paired seed). The B4 (tool-selection) floor of 0.000 is a corpus-density artifact, not a capacity gate: rebalancing the SFT mixture to tool-use ratio 1:21 yields VectraYX-Nano v7, the released headline configuration, reaching B4 = 0.230 +/- 0.052 at 42M while retaining B1 = 0.332 +/- 0.005 and B5 = 0.725 +/- 0.130; a LoRA replication on a 260M from-scratch mid-tier reaches 0.445 +/- 0.201. The released GGUF is 96 MB in F16, runs sub-second TTFT on commodity hardware under llama.cpp, and is, to our knowledge, the first published Spanish-native cybersecurity LLM with end-to-end MCP integration.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。