评测大模型在表格隐含预测任务中的推理能力。
TopBench: A Benchmark for Implicit Predictive Reasoning in Tabular Question Answering
- 构建包含779个样本的综合评测基准，涵盖多类隐含预测任务。
- 现有模型普遍误判意图，依赖简单查表而非深度推理。
- 准确识别用户意图是实现精准预测的关键前提。
大语言模型（LLMs）已显著提升表格问答性能，但多数查询仅需信息提取或简单聚合。现实中大量真实问题属于隐含预测类，需从历史模式中推断未观测答案，而非直接检索。此类问题带来双重挑战：识别潜在意图，以及在海量表格上进行可靠预测推理。为此，我们提出TopBench，一个包含779个样本的基准，覆盖四类子任务：单点预测、决策判断、治疗效应分析与复杂筛选，要求模型生成包含推理文本和结构化表格的输出。我们在文本式与代理式工作流下评估多种模型。实验表明，当前模型常因意图识别失败而退化为查表行为。深入分析发现，准确意图消歧是实现高级预测行为的前提。此外，提升预测精度上限需引入更复杂的建模或推理机制。
原文摘要 · Abstract (English)
Large Language Models (LLMs) have advanced Table Question Answering, where most queries can be answered by extracting information or simple aggregation. However, a common class of real-world queries is implicitly predictive, requiring the inference of unobserved answers from historical patterns rather than mere retrieval. These queries introduce two challenges: recognizing latent intent and reliable predictive reasoning over massive tables. To assess LLMs in such Tabular questiOn answering with implicit Prediction tasks, we introduce TopBench, a benchmark consisting of 779 samples across four sub-tasks, ranging from single-point prediction to decision making, treatment effect analysis, and complex filtering, requiring models to generate outputs spanning reasoning text and structured tables. We evaluate diverse models under both text-based and agentic workflows. Experiments reveal that current models often struggle with intent recognition, defaulting to just lookups. Deeper analysis identifies that accurate intent disambiguation serves as the prerequisite for leading these predictive behaviors. Furthermore, elevating the upper bound of prediction precision requires the integration of more sophisticated modeling or reasoning capabilities.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。