用用户数据蒸馏小模型，让大模型函数调用快78%。
ODIA: Oriented Distillation for Inline Acceleration of LLM-based Function Calling
- 从线上流量自动识别简单查询，蒸馏大模型知识到小模型
- 响应延迟降低45%（预期）和78%（中位数），准确率几乎不变
- 无需人工干预，持续更新，适合生产环境快速部署
函数调用是大语言模型（LLM）通过API与外部系统交互的关键技术。然而，基于LLM的函数调用存在高延迟问题，严重影响用户体验。本文提出一种名为面向内联加速的定向蒸馏（ODIA）的新方法，利用在线用户交互数据加速函数调用。通过自动识别生产流量中的“简单查询”，将大模型的知识蒸馏到更小的模型中，使响应延迟降低45%（预期）和78%（中位数），同时保持准确率。我们在一款音乐应用中真实部署该方法，发现小模型可处理60%的流量，且准确率损失微乎其微。该方法仅需极少人工干预，通过自动化数据收集与模型更新实现持续优化，为生产环境提供了实用解决方案。
原文摘要 · Abstract (English)
Function Calling is a crucial technique that enables Large Language Models (LLMs) to interact with external systems through APIs. However, the high latency associated with LLM-based Function Calling significantly impacts user experience. This paper presents a novel approach called Oriented Distillation for Inline Acceleration (ODIA) that leverages online user interaction data to accelerate Function Calling. By automatically identifying "simple queries" from production traffic and distilling knowledge from larger models to smaller ones, our method reduces response latency by 45% (expected) and 78% (median) while maintaining accuracy. We demonstrate the effectiveness of our approach through real-world deployment in a music application, where the smaller model successfully handles 60% of traffic with negligible accuracy loss. Our method requires minimal human intervention and continuously improves through automated data collection and model updating, making it a practical solution for production environments.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。