首个系统评估大模型预测能力的基准，发现其仍存显著短板。
Large Language Models for Predictive Analysis: How Far Are They?
- 构建包含1130个真实问题的PredictiQ基准，覆盖8个领域。
- 12个主流大模型在文本分析与代码生成上表现参差，准确率普遍不足。
- 适合关注大模型实际应用局限的研究者与开发者参考。
预测分析是现代决策的核心，广泛应用于多个领域。大语言模型（LLMs）因其强大的知识推理能力，在复杂决策任务中展现出潜力。然而，现有研究缺乏对LLMs在预测分析中能力的系统评估。为此，我们提出了PredictiQ基准，整合来自8个不同领域的44个真实数据集中的1130个复杂预测分析问题。设计了涵盖文本分析、代码生成及其对齐的评估协议，对12个知名LLMs进行了评估，揭示了当前模型在预测分析任务中的实际局限性。结果表明，现有大模型在该任务上仍面临显著挑战。相关代码与数据已开源于GitHub。
原文摘要 · Abstract (English)
Predictive analysis is a cornerstone of modern decision-making, with applications in various domains. Large Language Models (LLMs) have emerged as powerful tools in enabling nuanced, knowledge-intensive conversations, thus aiding in complex decision-making tasks. With the burgeoning expectation to harness LLMs for predictive analysis, there is an urgent need to systematically assess their capability in this domain. However, there is a lack of relevant evaluations in existing studies. To bridge this gap, we introduce the \textbf{PredictiQ} benchmark, which integrates 1130 sophisticated predictive analysis queries originating from 44 real-world datasets of 8 diverse fields. We design an evaluation protocol considering text analysis, code generation, and their alignment. Twelve renowned LLMs are evaluated, offering insights into their practical use in predictive analysis. Generally, we believe that existing LLMs still face considerable challenges in conducting predictive analysis. See \href{https://github.com/Cqkkkkkk/PredictiQ}{Github}.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。