首个量化大模型能源分析推理可靠性的评估框架，解决逻辑是否正确的问题。
Benchmarking Reasoning Reliability in Artificial Intelligence Models for Energy-System Analysis
- 构建五维指标体系，从逻辑、政策、不确定性等角度评估模型推理能力。
- 实测四大模型在能源场景下推理可靠性差异显著，顶级模型得分超90。
- 适合关注AI决策可信度的研究者与能源政策制定者参考。
人工智能和机器学习在能源领域的预测、优化与政策设计中日益普及，但缺乏标准化框架来评估其推理是否正确。现有验证主要关注预测准确率或计算效率，未检验分析结论的逻辑完整性。本研究提出分析可靠性基准（Analytical Reliability Benchmark, ARB），一个可复现的框架，用于量化大语言模型在能源系统分析中的推理可靠性。该基准整合五项子指标：准确性、推理可靠性、不确定性管理、政策一致性与透明性，基于开放的技术经济数据集（NREL ATB 2024、DOE H2A/H2New、IEA WEO 2024）在确定性、概率性和认知性场景下评估模型表现。测试了四个前沿模型（GPT-4/5、Claude 4.5 Sonnet、Gemini 2.5 Pro、Llama 3 70B），在相同事实与法规条件下进行对比。结果显示，推理可靠性可被客观测量；GPT-4/5 和 Claude 4.5 Sonnet 实现一致且符合政策的推理（分析可靠性指数 >90），Gemini 2.5 Pro 表现中等稳定，而 Llama 3 70B 未达专业门槛。统计验证确认差异显著且可复现。ARB 在能源文献中首次提供对因果、概率与政策驱动推理的定量验证方法，为全球能源转型中的可信、透明分析应用提供参考框架。
原文摘要 · Abstract (English)
Artificial intelligence and machine learning are increasingly used for forecasting, optimization, and policy design in the energy sector, yet no standardized framework exists to evaluate whether these systems reason correctly. Current validation practices focus on predictive accuracy or computational efficiency, leaving the logical integrity of analytical conclusions untested. This study introduces the Analytical Reliability Benchmark (ARB), a reproducible framework that quantifies reasoning reliability in large language models applied to energy system analysis. The benchmark integrates five submetrics: accuracy, reasoning reliability, uncertainty discipline, policy consistency, and transparency, and evaluates model performance across deterministic, probabilistic, and epistemic scenarios using open technoeconomic datasets (NREL ATB 2024, DOE H2A/H2New, IEA WEO 2024). Four frontier models (GPT-4/5, Claude 4.5 Sonnet, Gemini 2.5 Pro, Llama 3 70B) were tested under identical factual and regulatory conditions. Results show that reasoning reliability can be objectively measured. GPT-4/5 and Claude 4.5 Sonnet achieved consistent and policy-compliant reasoning (Analytical Reliability Index greater than 90), Gemini 2.5 Pro demonstrated moderate stability, and Llama 3 70B remained below professional thresholds. Statistical validation confirmed that these differences are significant and reproducible. The ARB establishes the first quantitative method in the energy literature for verifying causal, probabilistic, and policy-driven reasoning in artificial intelligence systems, providing a reference framework for trustworthy and transparent analytical applications in the global energy transition.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。