arXiv:2501.02342cs.LGcs.AI2025-01被引 6

用小模型实现车机功能调用,省去规则系统,实时响应。

Optimizing Small Language Models for In-Vehicle Function-Calling

  • 压缩Phi-3 mini模型,剪枝+量化,删20亿参数仍可用
  • 在无硬件加速下实现11词/秒生成速度,支持实时推理
  • 适合想部署轻量语言模型的车载系统开发者

我们提出一种全面方法,将小型语言模型(SLMs)作为车载边缘设备中的功能调用代理部署,替代传统基于规则的系统,提供更灵活、更稳健的方案。通过使用SLMs,简化车辆控制机制并提升用户体验。考虑到车载硬件限制,采用先进的模型压缩技术,包括结构化剪枝、修复和量化,确保模型在资源受限条件下仍保持可接受性能。本研究聚焦优化代表性SLM——微软Phi-3 mini,并总结嵌入式模型部署的最佳实践,涵盖压缩、任务特定微调及车辆集成。结果显示,尽管模型参数减少达20亿,该方法仍能准确高效地处理复杂车载任务。此外,在轻量级运行时环境中执行,实现11词/秒的生成速度,使无需硬件加速的实时本地推理成为可能。结果表明,SLMs有潜力重塑车载控制系统,实现用户与车辆间更直观的交互,提升驾驶体验。

原文摘要 · Abstract (English)

We propose a holistic approach for deploying Small Language Models (SLMs) as function-calling agents within vehicles as edge devices, offering a more flexible and robust alternative to traditional rule-based systems. By leveraging SLMs, we simplify vehicle control mechanisms and enhance the user experience. Given the in-vehicle hardware constraints, we apply state-of-the-art model compression techniques, including structured pruning, healing, and quantization, ensuring that the model fits within the resource limitations while maintaining acceptable performance. Our work focuses on optimizing a representative SLM, Microsoft's Phi-3 mini, and outlines best practices for enabling embedded models, including compression, task-specific fine-tuning, and vehicle integration. We demonstrate that, despite significant reduction in model size which removes up to 2 billion parameters from the original model, our approach preserves the model's ability to handle complex in-vehicle tasks accurately and efficiently. Furthermore, by executing the model in a lightweight runtime environment, we achieve a generation speed of 11 tokens per second, making real-time, on-device inference feasible without hardware acceleration. Our results demonstrate the potential of SLMs to transform vehicle control systems, enabling more intuitive interactions between users and their vehicles for an enhanced driving experience.

小模型车机系统边缘计算函数调用

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。