用持续预训练让小模型掌握嵌入式代码生成，效果超越大模型。
H2LooP Spark Preview: Continual Pretraining of Large Language Models for Low-Level Embedded Systems Code
- 基于硬件数据手册构建100B tokens嵌入式数据集，用高秩LoRA持续训练
- 在13个嵌入式领域上，困惑度降低超66%，代码补全准确率超多个大模型
- 适合做嵌入式开发的开源小模型开发者，或想低成本部署专业代码能力者
大型语言模型在通用编程中表现优异，但在低级嵌入式系统编程领域仍受限。该领域涉及硬件寄存器操作、厂商专用SDK、实时操作系统API和硬件抽象层，这些内容在标准预训练语料中严重缺失。本文提出H2LooP Spark Preview，一个持续预训练（CPT）流程，将完全开源的OLMo-3-7B模型适配嵌入式领域，采用BF16 LoRA与秩稳定缩放，在8块NVIDIA H100 GPU上训练。训练语料源自117家厂商的仓库-数据手册对，共100B原始嵌入式数据，通过SpecMap（Nipane等，2026）提出的分层数据手册-代码映射方法处理，最终得到包含23.5B token的精选数据集，覆盖13个嵌入式领域。使用高秩LoRA（r=512）进行持续预训练，使域内困惑度下降70.4%，外部仓库困惑度下降66.1%。在涵盖13个嵌入式领域的生成式代码补全基准测试中，该7B模型在8个类别上超过Claude Opus 4.6和Qwen3-Coder-30B的标记准确率。我们已在Huggingface发布生产级训练检查点作为开源资源。
原文摘要 · Abstract (English)
Large language models (LLMs) demonstrate strong code generation abilities in general-purpose programming languages but remain limited in specialized domains such as low-level embedded systems programming. This domain involves hardware register manipulation, vendor-specific SDKs, real-time operating system APIs, and hardware abstraction layers that are underrepresented in standard pretraining corpora. We introduce H2LooP Spark Preview, a continual pretraining (CPT) pipeline that adapts the OLMo-3-7B-a fully open language model to the embedded systems domain using BF16 LoRA with rank-stabilized scaling on 8 NVIDIA H100 GPUs. Our training corpus is constructed from repository-datasheet pairs covering 100B tokens of raw embedded systems data across 117 manufacturers, processed using the hierarchical datasheet-to-code mapping approach proposed in SpecMap (Nipane et al., 2026). The resulting curated dataset split contains 23.5B tokens across 13 embedded domains. Continual pretraining with high-rank LoRA (r=512) yields substantial gains, reducing in-domain perplexity by 70.4% and held-out repository perplexity by 66.1%. On generative code completion benchmarks spanning 13 embedded domains, our 7B model outperforms Claude Opus 4.6 and Qwen3-Coder-30B on 8 categories in token accuracy, showing that targeted continual pretraining enables smaller open-weight models to rival frontier systems on specialized technical tasks. We release the production training checkpoint on Huggingface as an open-source artifact.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。