10亿参数手机端大模型，长文本处理强且量化后性能稳。
MobileLLM-Pro Technical Report
- 用知识蒸馏让小模型学会长文本理解能力
- 支持12.8万词上下文，4比特量化仅轻微降效
- 适合移动端部署，开源模型代码供研究
约10亿参数的高效设备端语言模型对于实现移动和可穿戴设备上的低延迟AI应用至关重要。然而，在保持强大性能的同时支持长上下文窗口并实现实际部署仍面临重大挑战。我们提出MobileLLM-Pro，一个专为设备端部署优化的10亿参数语言模型。该模型在11项标准基准测试中达到当前最佳表现，显著优于Gemma 3-1B和Llama 3.2-1B，支持高达128,000个词的上下文窗口，并在4比特量化下仅出现轻微性能下降。这些改进得益于四项核心技术：(1) 隐式位置蒸馏，通过知识蒸馏有效赋予模型长上下文能力；(2) 专家模型融合框架，将多个领域专家合并为紧凑模型而不增加参数；(3) 基于效用估计的模拟数据混合；(4) 支持自蒸馏的4比特量化感知训练。我们已公开模型权重与代码，以推动高效设备端语言模型的研究。
原文摘要 · Abstract (English)
Efficient on-device language models around 1 billion parameters are essential for powering low-latency AI applications on mobile and wearable devices. However, achieving strong performance in this model class, while supporting long context windows and practical deployment remains a significant challenge. We introduce MobileLLM-Pro, a 1-billion-parameter language model optimized for on-device deployment. MobileLLM-Pro achieves state-of-the-art results across 11 standard benchmarks, significantly outperforming both Gemma 3-1B and Llama 3.2-1B, while supporting context windows of up to 128,000 tokens and showing only minor performance regressions at 4-bit quantization. These improvements are enabled by four core innovations: (1) implicit positional distillation, a novel technique that effectively instills long-context capabilities through knowledge distillation; (2) a specialist model merging framework that fuses multiple domain experts into a compact model without parameter growth; (3) simulation-driven data mixing using utility estimation; and (4) 4-bit quantization-aware training with self-distillation. We release our model weights and code to support future research in efficient on-device language models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。