针对手机端部署，设计了高效小模型PhoneLM。
PhoneLM:an Efficient and Capable Small Language Model Family through Principled Pre-training
- 先优化模型运行效率再预训练，提升设备适配性。
- 0.5B和1.5B版本达同类模型最优性能-效率平衡。
- 开源完整训练数据与代码，支持安卓指令调用演示。
小型语言模型（SLM）在设备端部署的需求日益增长，但现有设计往往忽略硬件特性。本文提出一项简单而有效的小模型设计原则：在预训练前进行架构搜索，以获得近似最优的运行效率。基于该原则，我们构建了PhoneLM SLM系列（当前包含0.5B和1.5B两个版本），在相似参数规模下实现了最先进的能力-效率权衡。我们完全开源PhoneLM的代码、权重及训练数据集，涵盖基础版与指令微调版。此外，还发布了可精准调用Android Intent的微调版本，并提供端到端安卓演示。所有资源均在https://github.com/UbiquitousLearning/PhoneLM公开获取。
原文摘要 · Abstract (English)
The interest in developing small language models (SLM) for on-device deployment is fast growing. However, the existing SLM design hardly considers the device hardware characteristics. Instead, this work presents a simple yet effective principle for SLM design: architecture searching for (near-)optimal runtime efficiency before pre-training. Guided by this principle, we develop PhoneLM SLM family (currently with 0.5B and 1.5B versions), that acheive the state-of-the-art capability-efficiency tradeoff among those with similar parameter size. We fully open-source the code, weights, and training datasets of PhoneLM for reproducibility and transparency, including both base and instructed versions. We also release a finetuned version of PhoneLM capable of accurate Android Intent invocation, and an end-to-end Android demo. All materials are available at https://github.com/UbiquitousLearning/PhoneLM.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。