OptGear实现超高效边缘端推理，支持长上下文与低内存占用。
Opt.Gear Technical Report
- 采用卷积-注意力混合架构，降低长序列KV缓存开销。
- 在NPU上推理速度比同规模模型快4.9倍，1B模型支持64K上下文。
- 首个可在MCU上达到20TPS的生成式语言模型，适合嵌入式部署。
我们提出OptGear，一个专为高效设备端部署、实时推理和强任务能力设计的基础模型。包含密集型模型（1M、270M、1B），上下文长度达64K。设计了一种新型混合架构，结合卷积键值门控混合器与局部全局注意力，有效降低随长上下文指数增长的KV缓存内存。该架构使预填充和解码速度在NPU上相比同规模模型最高提升4.9倍。从2万亿词候选语料中，仅使用0.5万亿词子集训练，无需知识蒸馏，是现有基础模型中最数据高效的。所有模型均开放权重及ONNX、高通NPU、苹果ANE部署二进制文件，适用于需快速、内存高效推理的边缘应用。为进一步拓展设备端生成式语言模型生态，我们推出OptGear-1M，即小型语言模型（TLM），可在微控制器单元（MCU）部署。OptGear-1M是首个在STM32H747I-DISCO的ARM Cortex-M7 CPU上以W4A32量化实现20TPS的生成式语言模型。
原文摘要 · Abstract (English)
We introduce OptGear, a foundation model designed for efficient on-device deployment, real-tim inference, and strong task capability. It includes a dense model (1M, 270M, and 1B) with a context length of 64K. We designed a new hybrid architecture that combines a convolutional key-value gated mixer with local-global attention to reduce the KV-cache memory that tends to increase exponentially with long context. This architecture delivers up to X4.9 faster prefill and decoding speeds on the NPUs compared to models of a similar scale models. From a 2T tokens candidate corpus, OptGear is trained on a curated 0.5T tokens subset without knowledge distillation. This is the most data-efficient of the existing foundation models. All models are released with open weights and deployment binaries for ONNX, Qualcomm NPU, and Apple ANE making OptGear a practical base for edge applications that need fast, memory-efficient inference and strong task capabilities. Furthermore, to expand the ecosystem of on-device generative language models, we are introducing the OptGear-1M that can be deployed on Micro-Controller Units (MCUs), a Tiny Language Model (TLM). OptGear-1M is the first generative language model to achieve 20 TPS with W4A32 quantization on the ARM Cortex-M7 CPU of the STM32H747I-DISCO.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。