研究发现大模型数学推理有单提示上限，60%-79%难突破。
Less Is More: Cognitive Load and the Single-Prompt Ceiling in LLM Mathematical Reasoning
- 通过40多种提示工程尝试，发现性能在约60%-79%饱和
- 最优提示（2252字节）在硬测试集上达79.25%准确率
- 模型注意力与提示顺序的非单调交互是主要瓶颈
我们对SAIR等式理论阶段1竞赛中的形式数学推理进行了系统性实证研究。任务要求判断一个等式法则是否在所有幺半群上蕴含另一个——一般情况不可判定，但可通过对有限模型搜索判定为假。五周内，我们设计、测试并分析了超过40种提示变体（0至4878字节），涵盖四个评估划分和三种模型（gpt-oss-120b、Llama 3.3 70B、Gemma 4 31B）。核心发现是单提示天花板：尽管投入大量工程努力，gpt-oss-120b在硬任务上的平衡准确率在60%-79%间趋于饱和，高于无提示基线（59.75%）。我们识别出三个机制：(1) 真命题的数学不可判定性限制了任何有限提示的表达能力；(2) 复杂规则系统会降低弱模型表现（如Llama 3.3 70B在提示超2KB时真值召回降为0%）；(3) 提示顺序与模型注意力产生脆弱且非单调的交互。最佳提交（AN45c，2252字节）在hard3（n=400）上达到79.25%准确率（95%置信区间：[75.0%, 82.9%]），真值召回95.9%，假值召回63.4%，相比基线提升19.5个百分点。所有提示变体、评估脚本与结果已开源。
原文摘要 · Abstract (English)
We present a systematic empirical study of prompt engineering for formal mathematical reasoning in the context of the SAIR Equational Theories Stage 1 competition. The task requires deciding whether one equational law implies another over all magmas -- a problem that is undecidable in general but decidable for FALSE via finite model search. Over five weeks, we designed, tested, and analyzed more than 40 prompt variants, ranging from 0 to 4,878 bytes, across four evaluation splits and three language models (gpt-oss-120b, Llama 3.3 70B, Gemma 4 31B). Our central finding is a single-prompt ceiling: despite substantial engineering effort, balanced hard accuracy plateaus in an empirical saturation region of approximately 60--79% for gpt-oss-120b, compared to a 59.75% no-cheatsheet baseline. We identify three mechanisms underlying this ceiling: (1) the mathematical undecidability of the TRUE case limits what any finite prompt can encode; (2) complex rule systems decrease performance on weaker models (Llama 3.3 70B collapses to 0% TRUE recall with prompts exceeding 2KB); and (3) prompt ordering effects interact with model attention in fragile, non-monotonic ways. Our best submission (AN45c, 2,252 bytes) achieves 79.25% accuracy on hard3 (n=400; 95% CI: [75.0%, 82.9%]), with TRUE recall of 95.9% and FALSE recall of 63.4%, representing a +19.5 percentage-point improvement over the no-cheatsheet baseline (59.75%). We release all prompt variants, evaluation scripts, and results at https://github.com/israelcazares/sair-prompt-engineering
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。