小模型也能高效运行在边缘设备上，本地推理更安全更快。
TinyLLM: A Framework for Training and Deploying Language Models at the Edge Computers
- 用30-120M参数的小模型替代大模型，通过精选数据提升性能。
- 实测小模型可在边缘设备上实现高吞吐率与准确率。
- 框架支持定制化训练与本地部署，适合隐私敏感场景。
语言模型因通用能力受到广泛关注，但其规模越大对计算资源要求越高，导致在移动和边缘设备上推理困难，通常需依赖远程服务，带来延迟、网络不稳定和隐私风险。为此，我们探索了反向路径：假设约30-120M参数的小模型可通过精心设计的预训练和微调数据，在特定任务上超越大模型。我们在支持感知应用的边缘模型部署背景下进行系统研究，训练多个基础模型发现，小模型可本地运行，实现高令牌速率与高准确率。基于此，我们构建了一个框架，支持用户为特定应用训练定制化基础模型并部署至边缘端。
原文摘要 · Abstract (English)
Language models have gained significant interest due to their general-purpose capabilities, which appear to emerge as models are scaled to increasingly larger parameter sizes. However, these large models impose stringent requirements on computing systems, necessitating significant memory and processing requirements for inference. This makes performing inference on mobile and edge devices challenging, often requiring invocating remotely-hosted models via network calls. Remote inference, in turn, introduces issues like latency, unreliable network connectivity, and privacy concerns. To address these challenges, we explored the possibility of deviating from the trend of increasing model size. Instead, we hypothesize that much smaller models (~30-120M parameters) can outperform their larger counterparts for specific tasks by carefully curating the data used for pre-training and fine-tuning. We investigate this within the context of deploying edge-device models to support sensing applications. We trained several foundational models through a systematic study and found that small models can run locally on edge devices, achieving high token rates and accuracy. Based on these findings, we developed a framework that allows users to train foundational models tailored to their specific applications and deploy them at the edge.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。