本地大模型在复杂编程题上表现有限，仅达商用模型一半准确率。
Evaluating the Limitations of Local LLMs in Solving Complex Programming Challenges
- 构建离线评估框架，支持多日运行中断续
- 在3589道题上测试8个6.7-9亿参数模型，平均通过率不足商用模型一半
- 适合关注开源模型实际能力的开发者与企业自研团队
本研究评估当前开源本地部署大语言模型在处理具有长描述和复杂上下文的竞赛编程任务中的表现。基于原始的AI驱动代码生成评估框架（FACE），作者改造其流程以完全离线运行，通过Ollama运行时将原框架庞大的按题目录结构压缩为少量合并的JSON文件，并加入可靠断点机制，支持多日运行中断后恢复。该增强框架对包含3,589道题的Kattis题库，使用8个参数量在6.7至90亿之间的代码导向型模型进行生成、提交与记录。结果显示，本地模型的整体pass@1准确率较低，最佳模型的通过率仅为专有模型Gemini 1.5和ChatGPT-4的一半。这些发现揭示了私有可控部署与顶尖专有服务之间仍存在显著差距，但也凸显了开源模型的快速进步以及该可复现评估流程在企业内设硬件上的实用价值。
原文摘要 · Abstract (English)
This study examines the performance of today's open-source, locally hosted large-language models (LLMs) in handling complex competitive programming tasks with extended problem descriptions and contexts. Building on the original Framework for AI-driven Code Generation Evaluation (FACE), the authors retrofit the pipeline to work entirely offline through the Ollama runtime, collapsing FACE's sprawling per-problem directory tree into a handful of consolidated JSON files, and adding robust checkpointing so multi-day runs can resume after failures. The enhanced framework generates, submits, and records solutions for the full Kattis corpus of 3,589 problems across eight code-oriented models ranging from 6.7-9 billion parameters. The submission results show that the overall pass@1 accuracy is modest for the local models, with the best models performing at approximately half the acceptance rate of the proprietary models, Gemini 1.5 and ChatGPT-4. These findings expose a persistent gap between private, cost-controlled LLM deployments and state-of-the-art proprietary services, yet also highlight the rapid progress of open models and the practical benefits of an evaluation workflow that organizations can replicate on in-house hardware.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。