测试大模型在真实复杂表格中的推理能力，发现现有模型表现不佳。
RUST-BENCH: Benchmarking LLM Reasoning on Unstructured Text within Structured Tables
- 构建包含2031张真实表格的基准测试集，覆盖科学与体育领域。
- 模型在跨数千词的多跳推理中准确率不足，尤其面对异构字段时。
- 适合研究表格推理、大模型泛化能力的学者与开发者参考。
现有表格推理基准大多针对小规模、同质化表格，无法反映真实数据的复杂性，也未能全面评估大语言模型（LLMs）的推理能力。真实表格往往长度长、结构异构且领域特定，混合了结构化字段与自由文本，需跨数千个标记进行多跳推理。为填补这一空白，我们提出RUST-BENCH，一个涵盖2031张真实世界表格、7966个问题的基准测试，覆盖两个领域：i) RB-Science（NSF资助记录）和ii) RB-Sports（NBA统计数据）。与以往工作不同，RUST-BENCH联合评估模型在规模、异构性、领域特异性及推理复杂度上的表现。对开源与专有模型的实验表明，当前模型在处理异构模式和复杂多跳推理时仍存在显著缺陷，揭示了现有架构与策略的持续弱点。RUST-BENCH为推进表格推理研究设立了新的挑战性测试平台。
原文摘要 · Abstract (English)
Existing tabular reasoning benchmarks mostly test models on small, uniform tables, underrepresenting the complexity of real-world data and giving an incomplete view of Large Language Models' (LLMs) reasoning abilities. Real tables are long, heterogeneous, and domain-specific, mixing structured fields with free text and requiring multi-hop reasoning across thousands of tokens. To address this gap, we introduce RUST-BENCH, a benchmark of 7966 questions from 2031 real-world tables spanning two domains: i) RB-Science (NSF grant records) and ii) RB-Sports (NBA statistics). Unlike prior work, RUST-BENCH evaluates LLMs jointly across scale, heterogeneity, domain specificity, and reasoning complexity. Experiments with open-source and proprietary models show that LLMs struggle with heterogeneous schemas and complex multi-hop inference, revealing persistent weaknesses in current architectures and prompting strategies. RUST-BENCH establishes a challenging new testbed for advancing tabular reasoning research.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。