将大模型函数调用能力高效迁移至超小模型,性能超越同尺寸开源模型。
STAR: Similarity-guided Teacher-Assisted Refinement for Super-Tiny Function Calling Models
- 通过约束知识蒸馏稳定训练,抑制错误预测,保留探索能力。
- 引入相似度感知强化学习,用细粒度连续奖励提升策略优化。
- 0.6B模型在10亿参数以下模型中表现最佳,适合资源受限场景部署。
大型语言模型(LLMs)在函数调用中的应用对构建先进AI代理至关重要,但其规模过大限制了广泛使用,亟需将能力迁移到小型模型。现有方法常面临过拟合、训练不稳定、多解任务中二元奖励无效及技术协同困难等问题。本文提出STAR:相似度引导的教师辅助精炼框架,实现大模型能力向超小模型的有效迁移。STAR包含两项核心技术:(1) 约束知识蒸馏(CKD),在top-k前向KL散度基础上增强,抑制置信度高的错误预测,确保训练稳定的同时保留下游强化学习的探索能力;(2) 相似度引导强化学习(Sim-RL),引入基于细粒度相似度的奖励机制,通过评估生成输出与真实答案的相似性,提供鲁棒、连续且丰富的信号以优化策略。在多个挑战性基准上进行大量实验,验证了方法有效性。所提STAR模型在各自尺寸类别中达到最新技术水平,显著优于基线。尤为突出的是,0.6B的STAR模型在所有小于10亿参数的开源模型中表现最佳,超越多个更大规模的知名开源模型。STAR为将大模型能力蒸馏至超小模型提供了统一训练框架,推动强大、可访问且高效的AI代理发展。
原文摘要 · Abstract (English)
The proliferation of Large Language Models (LLMs) in function calling is pivotal for creating advanced AI agents, yet their large scale hinders widespread adoption, necessitating transferring their capabilities into smaller ones. However, existing paradigms are often plagued by overfitting, training instability, ineffective binary rewards for multi-solution tasks, and the difficulty of synergizing techniques. We introduce STAR: Similarity-guided Teacher-Assisted Refinement, a novel holistic framework that effectively transfers LLMs' capabilities to super-tiny models. STAR consists of two core technical innovations: (1) Constrained Knowledge Distillation (CKD), a training objective that augments top-k forward KL divergence to suppress confidently incorrect predictions, ensuring training stability while preserving exploration capacity for downstream RL. STAR holistically synergizes these strategies within a cohesive training curriculum, enabling super-tiny models to achieve exceptional performance on complex function calling tasks; (2) Similarity-guided RL (Sim-RL), a RL mechanism that introduces a fine-grained, similarity-based reward. This provides a robust, continuous, and rich signal for better policy optimization by evaluating the similarity between generated outputs and the ground truth. Extensive experiments on challenging and renowned benchmarks demonstrate the effectiveness of our method. Our STAR models establish SOTA in their size classes, significantly outperforming baselines. Remarkably, our 0.6B STAR model achieves the best performance among all open models under 1B, surpassing even several well-known open models at a larger scale. STAR demonstrates a training framework that distills capabilities of LLMs into super-tiny models, paving the way for powerful, accessible, and efficient AI agents.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。