评测小模型在边缘设备上执行工具调用的可靠性，发现大模型更稳定但资源消耗高。
Beyond Fluent Generation: A CPU Reliability Benchmark for MCP-Style Tool Calling in Sub-2B Small Language Models for Edge Deployment
- 构建跨平台CPU基准，测试5个20亿参数以下模型在100个任务中的工具调用表现。
- 最大模型Qwen2.5-1.5B在采样下达79%正确率，小模型0.5B版采样后骤降至32%。
- 强调输出修复和安全机制对边缘部署至关重要，适合关注边缘AI可靠性的研究者。
资源受限的单板计算机（如Raspberry Pi、NVIDIA Jetson Nano等）推动了本地化小语言模型（SLM）代理的发展，以减少云端依赖、提升数据本地性并应对网络中断。模型上下文协议（MCP）风格的工具调用不仅要求流畅生成，还需输出可机器解析的JSON，正确选择工具，完整提供参数，并避免意外行为。本文通过评估五个开源小模型（Phi-1.5、Pythia-1.4B、TinyLlama-1.1B-Chat、Qwen2.5-0.5B、Qwen2.5-1.5B）在100个提示下的表现，建立无平台依赖的CPU基准，涵盖天气查询、网页搜索、计算、邮件撰写与任务创建，使用贪婪解码和核采样。通过恢复解析器剔除Markdown标记，提取花括号子串，评估可解析性、工具名正确性、参数完整性与值一致性。结果显示：Qwen2.5-1.5B在贪婪解码下达75%，核采样下为79%；Qwen2.5-0.5B贪婪解码为72%，但核采样仅32%；Phi-1.5为0%，Pythia与TinyLlama最高仅7%。严格事后审计发现，1000条原始输出中仅有5条可直接解析为合法JSON，表明严重依赖输出修复。CPU资源探测显示，Qwen2.5-1.5B需7,960 MiB内存与30.782秒平均延迟，而0.5B版本仅需3,637 MiB与10.627秒，揭示边缘部署中可靠性与资源间的权衡。结果未覆盖具体开发板或完整MCP实现。安全部署需结合模式验证、约束生成、最小权限执行及关键操作的人工介入。
原文摘要 · Abstract (English)
Resource constrained single-board computers including Raspberry Pi, NVIDIA Jetson Nano, Arduino UNO Q, Orange Pi, and LattePanda motivate on-device small language model (SLM) agents that reduce cloud dependence, improve data locality, and tolerate intermittent connectivity. Model Context Protocol (MCP)-style tool invocation demands more than fluent generation: an agent must emit machine-readable JSON, select the correct tool, supply all required arguments, and avoid unintended actions. We establish a platform-agnostic CPU baseline by evaluating five open-weight models below two billion parameters Phi-1.5, Pythia-1.4B, TinyLlama-1.1B-Chat, Qwen2.5-0.5B, and Qwen2.5-1.5B on 100 prompts spanning weather retrieval, web search, calculation, email composition, and task creation, under greedy decoding and nucleus sampling. A recovery parser strips Markdown fences, extracts brace-delimited substrings, and scores parseability, tool-name correctness, argument completeness, and value agreement. Under this criterion, Qwen2.5-1.5B achieves 75% (greedy) and 79% (sampling); Qwen2.5-0.5B achieves 72% (greedy) but drops to 32% under sampling. Phi-1.5 scores 0%; Pythia and TinyLlama reach at most 7%. A strict post-hoc audit finds only 5 of 1,000 raw responses directly parseable as JSON, exposing near-total dependence on output recovery. A CPU resource probe shows Qwen2.5-1.5B requires 7,960 MiB and 30.782 s mean latency; Qwen2.5-0.5B uses 3,637 MiB and 10.627 s, revealing a reliability-resource trade-off for edge deployment. These results do not cover the named boards directly or a full MCP implementation. Safe deployment requires schema validation, constrained generation, least-privilege execution, and human escalation for consequential actions.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。