arXiv:2608.24145cs.CLcs.SE2026-08

评测大模型处理结构化数据时的可信度与鲁棒性,发现当前模型表现仍有巨大提升空间。

TrustDABench: Benchmarking Reliability and Robustness of LLMs for Structured Data Analysis

论文配图:TrustDABench: Benchmarking Reliability and Robustness of LLMs for Structured Data Analysis
图 1 · 摘自论文原文
  • 从证据路径出发,设计19种扰动方法构建评测基准
  • 最佳模型可靠性仅24.21%、鲁棒性仍存9.10%错误率
  • 适合关注大模型数据分析可信性的研究者和开发者

大模型被广泛用于分析表格、CSV等结构化数据,但看似正确的回答未必可信。可信结果需有从问题到数据证据的有效推理路径。为此提出两个诊断问题:模型能否在缺乏有效路径时拒绝回答或请求澄清;能否在数据表达形式变化时保持正确分析。本文提出TrustDABench基准,基于证据路径视角设计19种扰动算子,并通过代理型大模型生成框架实现。该基准包含2,340个经人工验证的扰动实例,评估8个代表性大模型。结果显示存在显著提升空间:最佳可靠性为GPT-5.5的24.21%平均MRS,最佳鲁棒性为Claude-Sonnet-5的9.10%平均ASR。模型失败具有系统性:难以识别冲突证据,常沿可执行但无支持的分析路径继续推演,对表间关系或观测边界变化敏感。这表明可靠的数据分析仍需更强的证据边界识别与表示不变推理能力。

原文摘要 · Abstract (English)

LLMs are increasingly used to analyze spreadsheets, CSV files, and other structured data, but producing a correct-looking answer is not the same as producing a trustworthy analysis. A trustworthy result should be supported by a valid path from the user question to the relevant data evidence. This requirement creates two diagnostic questions: whether an LLM can refuse to answer or ask for clarification when such a path does not exist, and whether it can preserve the correct analysis when the same evidence is expressed in different table forms. We introduce TrustDABench, a benchmark that operationalizes these questions as reliability and robustness. Starting from the evidence-path view, we derive 19 perturbation operators and instantiate them through an Agentic-LLM-based generation framework. TrustDABench contains 2,340 human-verified perturbed instances, and we evaluate eight representative LLMs. The results show substantial headroom: the best reliability result is only 24.21% average MRS, achieved by GPT-5.5, while the best robustness result still has 9.10% average ASR, achieved by Claude-Sonnet-5. The failures are systematic: models rarely detect conflicting evidence, often continue along executable but unsupported analysis paths, and remain sensitive to perturbations that change observation boundaries or cross-table relations. These findings suggest that stronger evidence-boundary recognition and representation-invariant reasoning are still needed for reliable structured-data analysis.

大模型评测结构化数据可信分析鲁棒性

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。