arXiv:2510.16943cs.LGcs.AI2025-10被引 2

通过细粒度评估,揭示大模型优化建模中的错误根源。

Peering Inside the Black Box: Uncovering LLM Errors in Optimization Modelling through Component-Level Evaluation

  • 按组件评估大模型生成的优化模型,细分变量与约束表现。
  • GPT-5表现最优,思维链与模块化提示效果最好,低约束误差是关键。
  • 适合关注模型可靠性与效率的优化算法研究者和开发者。

大型语言模型(LLMs)正被用于将自然语言描述转化为数学优化模型。现有评估多以整体为单位,依赖解的准确性或运行时间等粗略指标,难以发现结构或数值错误。本文提出一种全面的组件级评估框架,引入决策变量与约束的精确率、召回率,以及目标函数和约束的均方根误差(RMSE),并结合令牌使用量与延迟等效率指标。我们在六种提示策略下,对GPT-5、LLaMA 3.1 Instruct和DeepSeek Math在不同复杂度的优化问题上进行评估。结果表明,GPT-5持续领先,思维链、自一致性及模块化提示最为有效。分析显示,求解器性能主要取决于高约束召回率与低约束RMSE,二者共同保障结构正确性与解的可靠性。约束精确率与决策变量指标作用较小,而简洁输出则提升计算效率。研究提出三个NLP转优化建模原则:(i) 完整覆盖约束以防违规;(ii) 最小化约束RMSE以保证求解精度;(iii) 输出简洁以提升效率。该框架为优化建模中大模型的细粒度诊断评估奠定基础。

原文摘要 · Abstract (English)

Large language models (LLMs) are increasingly used to convert natural language descriptions into mathematical optimization formulations. Current evaluations often treat formulations as a whole, relying on coarse metrics like solution accuracy or runtime, which obscure structural or numerical errors. In this study, we present a comprehensive, component-level evaluation framework for LLM-generated formulations. Beyond the conventional optimality gap, our framework introduces metrics such as precision and recall for decision variables and constraints, constraint and objective root mean squared error (RMSE), and efficiency indicators based on token usage and latency. We evaluate GPT-5, LLaMA 3.1 Instruct, and DeepSeek Math across optimization problems of varying complexity under six prompting strategies. Results show that GPT-5 consistently outperforms other models, with chain-of-thought, self-consistency, and modular prompting proving most effective. Analysis indicates that solver performance depends primarily on high constraint recall and low constraint RMSE, which together ensure structural correctness and solution reliability. Constraint precision and decision variable metrics play secondary roles, while concise outputs enhance computational efficiency. These findings highlight three principles for NLP-to-optimization modeling: (i) Complete constraint coverage prevents violations, (ii) minimizing constraint RMSE ensures solver-level accuracy, and (iii) concise outputs improve computational efficiency. The proposed framework establishes a foundation for fine-grained, diagnostic evaluation of LLMs in optimization modeling.

大模型评估优化建模组件级分析

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。