arXiv:2504.19277cs.AIcs.SE2025-04中稿 · EASE 2025 AI Model…被引 12

小模型也能做函数调用,但格式生成仍不稳。

Small Models, Big Tasks: An Exploratory Empirical Study on Small Language Models for Function Calling

  • 对比零样本、少样本和微调,验证小模型在函数调用中的表现
  • 微调后准确率最高,但输出格式遵循率仍低
  • 适合边缘设备部署,尤其关注低延迟与内存使用场景

函数调用是信息检索、软件工程和自动化等领域的复杂任务。例如,查询1月15日从纽约飞往伦敦的最短航班需正确识别参数以生成精准调用。大语言模型(LLMs)虽能自动化此过程,但计算开销大,在资源受限环境中不实用。相比之下,小语言模型(SLMs)运行高效,响应快且计算需求低,适合在边缘设备上执行函数调用。本文开展探索性实证研究,评估SLMs在多个领域中通过零样本、少样本及微调方法生成函数调用的效果,涵盖有无提示注入的情形,并提供微调后的模型以支持后续应用。同时,我们从多个维度分析模型输出质量。此外,在边缘设备上测试其延迟与内存占用,揭示实际可行性。结果表明:尽管从零样本到少样本再到微调性能逐步提升,但模型在遵循输出格式方面仍存在显著问题;提示注入实验显示模型整体鲁棒,性能仅轻微下降。虽然SLMs在函数调用任务中展现潜力,但实时应用仍需进一步优化。

原文摘要 · Abstract (English)

Function calling is a complex task with widespread applications in domains such as information retrieval, software engineering and automation. For example, a query to book the shortest flight from New York to London on January 15 requires identifying the correct parameters to generate accurate function calls. Large Language Models (LLMs) can automate this process but are computationally expensive and impractical in resource-constrained settings. In contrast, Small Language Models (SLMs) can operate efficiently, offering faster response times, and lower computational demands, making them potential candidates for function calling on edge devices. In this exploratory empirical study, we evaluate the efficacy of SLMs in generating function calls across diverse domains using zero-shot, few-shot, and fine-tuning approaches, both with and without prompt injection, while also providing the finetuned models to facilitate future applications. Furthermore, we analyze the model responses across a range of metrics, capturing various aspects of function call generation. Additionally, we perform experiments on an edge device to evaluate their performance in terms of latency and memory usage, providing useful insights into their practical applicability. Our findings show that while SLMs improve from zero-shot to few-shot and perform best with fine-tuning, they struggle significantly with adhering to the given output format. Prompt injection experiments further indicate that the models are generally robust and exhibit only a slight decline in performance. While SLMs demonstrate potential for the function call generation task, our results also highlight areas that need further refinement for real-time functioning.

小模型函数调用边缘计算推理优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。