对比三款本地部署小模型在数学推理中的准确率与能效，发现高准确率未必更优。
More Accurate or More Efficient? Evaluating Locally Deployed Compact Open-Weight Language Models for Mathematical Reasoning
- 统一环境测试三款开源小模型，严格控制推理条件与评估流程。
- Gemma3:4b能效最高，每瓦时产出正确答案是Qwen3:4b的三倍。
- 仅看准确率会误导选择，能效与生成效率同样关键，适合本地部署者参考。
大型语言模型正越来越多地部署在本地设备上以保障隐私、降低开支并提升可访问性。然而，多数评估侧重准确性，而忽视了本地运行时长、能耗、失败模式分析及受控条件下的配对统计比较。本文提出一套受控、可复现的本地部署大模型数学推理评估流程，包含固定推理设置、分层答案提取与验证、显式失败模式分类以及逐题资源测量，并报告带配对显著性检验和效应量的准确率结果。我们在一台工作站上，使用同一推理服务器、共享提示模板与匹配题目集，对三款参数量低于五亿的紧凑型开源模型（Gemma3:4b、Phi3:3.8b、Qwen3:4b）在八年级数学、微积分I及高等概率统计三个数据集上进行了测试。结果显示无单一模型全面领先：Qwen3:4b在两个数据集上最准确，Gemma3:4b在微积分I上表现最佳；但Gemma3:4b每瓦时产生的正确答案数约为Qwen3:4b的三倍，且生成输出令牌更少；相比之下，Qwen3:4b耗时更长、能耗更高、输出更多。Phi3:3.8b在所有数据集上准确率显著更低，其低提取失败率表明错误主要源于内容而非格式解析问题，但可能受提示格式影响。初步结果表明，仅凭准确率无法有效指导本地模型选型。
原文摘要 · Abstract (English)
Large language models are increasingly deployed on local hardware for privacy, cost, and accessibility reasons. Yet many evaluations emphasize accuracy while fewer quantify local runtime and energy, characterize failure modes, or apply paired statistical comparisons under controlled conditions. This paper presents a controlled, documented procedure for evaluating locally hosted LLMs on mathematical reasoning. It combines fixed inference settings, hierarchical answer extraction and verification, explicit failure-mode classification, and per-question resource measurement, and reports accuracy with paired significance tests and effect sizes. We demonstrate it in a preliminary study of three compact open-weight models under five billion parameters, Gemma3:4b (Google), Phi3:3.8b (Microsoft), and Qwen3:4b (Alibaba), across datasets spanning Grade 8 Math, Calculus I, and Advanced Probability and Statistics. All models ran through the same local inference server on one workstation, using a shared prompt template, controlled settings, and a matched question set per dataset. No single model dominates. Qwen3:4b is most accurate on two datasets and Gemma3:4b on Calculus I, yet Gemma3:4b returns roughly three times more correct answers per watt-hour than Qwen3:4b on every dataset while generating far fewer output tokens; Qwen3:4b requires substantially more generation time, energy, and output per question. Phi3:3.8b is substantially less accurate on all three datasets; its low extraction-failure rate indicates incorrect answers rather than unparsed output, though we caveat possible prompt-format effects. These preliminary findings indicate that accuracy alone is an insufficient basis for selecting a local model.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。