评测大模型在编程竞赛题上的解题能力，发现性能差异显著。
LLM-ProS: Analyzing Large Language Models' Performance in Competitive Problem Solving
- 用166道国际编程竞赛真题测试大模型解题思路。
- GPT-4o和o1-preview在正确率上表现最优，但资源消耗高。
- 适合研究算法推理与模型评估的开发者参考。
大语言模型的快速发展为自动化复杂问题求解（如算法编程、程序竞赛）开辟了新路径。本文提出一种新型评估方法LLM-ProS，用于衡量先进大模型在国际大学生程序设计竞赛（ICPC）题目上的表现。基于2011至2024年世界总决赛的166道精选题目，我们对模型的推理能力、准确性和效率进行基准测试。评估涵盖GPT-4o、Mistral Large、Llama-3.1-405B以及o1系列（o1-mini和o1-preview）五款模型，关键指标包括正确率、资源利用效率和回答校准度。结果揭示模型在泛化、适应新题型和解决未见问题方面存在显著差异。同时研究了训练方法、数据污染及思维链推理对性能的影响。研究为优化大模型在算法任务中的表现提供了新洞见，明确指出了当前模型的优势与局限。
原文摘要 · Abstract (English)
The rapid advancement of large language models has opened new avenues for automating complex problem-solving tasks such as algorithmic coding and competitive programming. This paper introduces a novel evaluation technique, LLM-ProS, to assess the performance of state-of-the-art LLMs on International Collegiate Programming Contest (ICPC) problems. Using a curated dataset of 166 World Finals problems from 2011 to 2024, we benchmark the models' reasoning, accuracy, and efficiency. We evaluate the five models-GPT-4o, Mistral Large, Llama-3.1-405B, and the o1 family, consisting of o1-mini and o1-preview, across critical metrics like correctness, resource utilization, and response calibration. Our results reveal significant differences in the models' abilities to generalize, adapt, and solve novel problems. We also investigated the impact of training methodologies, dataset contamination, and chain-of-thought reasoning on model performance. The findings provide new insights into optimizing LLMs for algorithmic tasks, highlighting both strengths and limitations of current models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。