让大模型函数调用实时运行,速度提升6倍
SimpleTool: Parallel Decoding for Real-Time LLM Function Calling
- 用特殊标记压缩冗余符号并并行生成函数名与参数
- 实测速度提升3-6倍,4B模型延迟仅61.2ms
- 适合游戏AI、机器人控制等低延迟场景
基于大模型的函数调用使智能体能与外部工具交互,但自回归解码带来固有的延迟瓶颈,限制了具身智能、游戏AI和交互式虚拟人等实时应用(如10 Hz控制频率)。我们发现函数调用与自由文本生成本质不同:结构化输出存在大量令牌冗余(分隔符、参数名),且参数间因果依赖弱。关键在于协同利用这两点实现实时性能。提出SimpleTool,引入兼具双重功能的特殊标记:压缩低熵令牌(4-6倍减少),同时作为模式选择器,实现函数名与参数的独立并行生成。该协同设计在仅增加8.2%并行开销下,实现3-6倍端到端加速(最高达9.6倍)。在五个基准测试上对Qwen系列模型(0.5B-14B)的实验表明,显著提速的同时保持甚至提升准确性。在Mobile Actions数据集上,ST-Qwen-0.5B在准确率和延迟一致性上均优于Google FunctionGemma。在消费级GPU上量化部署后,实现61.2ms P50延迟,支持4B模型规模下16 Hz实时控制,弥合了大模型函数调用与真实世界低延迟部署之间的差距。
原文摘要 · Abstract (English)
LLM-based function calling enables intelligent agents to interact with external tools and environments, yet autoregressive decoding imposes a fundamental latency bottleneck that limits real-time applications such as embodied intelligence, game AI, and interactive avatars (e.g., 10 Hz control frequency). We observe that function calling differs fundamentally from free-form text generation: structured outputs exhibit substantial token redundancy (delimiters, parameter names), and arguments exhibit weak causal dependencies. Crucially, these two properties must be exploited jointly to achieve real-time performance. We present SimpleTool, which introduces special tokens that serve a dual role: compressing low-entropy tokens (4-6x reduction) while acting as mode selectors that enable independent parallel generation of function name and arguments. This synergistic design achieves 3-6x end-to-end speedup (up to 9.6x) with only +8.2% parallelization overhead. Experiments on five benchmarks across Qwen-series models (0.5B-14B) demonstrate substantial speedup while maintaining competitive or improved accuracy. On Mobile Actions, ST-Qwen-0.5B outperforms Google's FunctionGemma in both accuracy and latency consistency. With quantization on consumer-grade GPU, SimpleTool achieves 61.2ms P50 latency, enabling 16 Hz real-time control at 4B model scale, bridging the gap between LLM function calling and latency-critical real-world deployment.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。