大模型编程任务的准确率被高估，稳定性和重复运行可靠性需额外评估。
Accuracy, Stability, and Repeated-Run Reliability of Large Language Models on Deterministic Programming Tasks
- 设计重复调用评估协议，量化模型在相同任务下的输出稳定性。
- 准确率比无重试覆盖率最高高出17.8个百分点，且中等性能模型差距最大。
- 提示词效果因模型而异，不能一概而论，部署需关注稳定性。
在100道基于最近题目的LeetCode风格编程任务上，对5个厂商家族共16个大模型进行评估，采用两种提示模板、每题重复运行5次，共生成16,000个实例。标准的单次运行通过率会高估无重试覆盖率，最高达17.8个百分点；该差距在中等性能模型中最为显著。尽管运行级通过率与完全稳定性之间高度相关（r=0.985），但通过率始终高于无重试覆盖率，甚至导致模型排名反转。提示词影响具有模型依赖性，非普遍有益。结果表明，对确定性文本生成任务，重复运行稳定性分析是传统准确率报告的必要补充。
原文摘要 · Abstract (English)
Run-level pass rate overstates retry-free coverage by up to 17.8 percentage points -- and the gap is largest precisely for mid-performing systems. We investigate this accuracy--stability relationship in large language model (LLM) evaluation for deterministic text-conditioned generation, using programming tasks as a concrete testbed. Standard code-generation benchmarks emphasize single-run accuracy or eventual success under repeated sampling, but many deployment settings also require stability: consistent outcomes across repeated invocations under the same task description. We present a repeated-run evaluation protocol with metrics for run-level accuracy, retry-free coverage, and per-problem variability. On a recency-based benchmark of 100 LeetCode-style problems, we evaluate 16 models from five provider families under two prompt templates with five repeated runs per problem, yielding 16,000 evaluation instances. Although run-level pass rate and perfect stability rate are strongly correlated (r=0.985), pass rate consistently exceeds retry-free coverage -- a gap that reaches 17.8 percentage points and reverses model rankings even among closely matched systems. Prompt effects are model-dependent rather than uniformly beneficial. These results suggest that repeated-run stability analysis is a necessary complement to conventional accuracy reporting for deterministic text-conditioned generation tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。