不同表述方式影响大模型解难问题能力，真实场景题更难解。
A Knapsack by Any Other Name: Presentation impacts LLM performance on NP-hard problems
- 用自然语言构建真实生活中的难解优化问题数据集
- 课本题准确率远超生活化和反向规则题
- 即使推理模型也受表述影响，说明泛化能力弱
为探究问题表述对大模型求解优化问题能力的影响，我们构建了日常难解优化问题（EHOP）数据集，包含教科书式问题（如图着色）、生活化场景问题（如派对安排）以及规则反转的变体。实验发现，当前顶尖大模型在多种提示策略下，对教科书题的解答准确率显著高于生活化和规则反转版本。尽管推理型模型表现更好，但其结果在不同表述间波动剧烈，表明其缺乏真正稳健的推理机制。这说明大模型仍严重依赖训练中见过的模式，难以推广到新问题。
原文摘要 · Abstract (English)
To investigate the effect of problem presentation on LLMs' ability to solve optimization problems, we introduce the dataset of Everyday Hard Optimization Problems (EHOP), a collection of NP-hard problems expressed in natural language. EHOP includes problem formulations that could be found in computer science textbooks (e.g., graph coloring), versions that are dressed up as problems that could arise in real life (e.g., party planning), and variants with inverted rules. We find that state-of-the-art LLMs, across multiple prompting strategies, systematically solve textbook problems more accurately than their real-life and inverted counterparts. While reasoning models are more capable, they nonetheless show high variance across problem presentations, suggesting they lack a truly robust reasoning mechanism. We argue that this constitutes evidence that LLMs are still heavily dependent on what was seen in training and struggle to generalize to novel problems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。