测试大模型在长文档数据分析中的真实表现，发现现有模型仍有明显短板。
LongDA: Benchmarking LLM Agents for Long-Document Data Analysis
- 构建了包含505个真实分析任务的长文档基准数据集
- 模型需从杂乱文档中提取信息并生成可执行代码，准确率普遍不足60%
- 适合关注大模型实际应用能力的研究者和开发者
我们提出LongDA，一个针对大语言模型（LLM）代理在文档密集型数据分析工作流中表现的评测基准。与以往假设输入规范、结构清晰的评测不同，LongDA聚焦真实场景下长文档阅读与复杂数据处理的瓶颈问题。基于17个公开的美国国家级调查数据，我们人工整理原始数据文件、长篇且多样化的文档资料以及专家撰写的出版物，从中提取出505个源于实际分析实践的查询任务。解决这些任务要求代理首先从多个非结构化文档中检索并整合关键信息，再进行多步计算并生成可执行代码，这对现有数据分析师代理仍是巨大挑战。为此，我们开发了LongTA——一种支持文档访问、检索和代码执行的工具增强型代理框架，并评估了多种专有及开源模型。实验表明，即使是顶尖模型之间也存在显著性能差距，凸显了在高风险、现实决策支持场景中应用LLM代理前需重视的诸多挑战。
原文摘要 · Abstract (English)
We introduce LongDA, a data analysis benchmark for evaluating LLM-based agents under documentation-intensive analytical workflows. In contrast to existing benchmarks that assume well-specified schemas and inputs, LongDA targets real-world settings in which navigating long documentation and complex data is the primary bottleneck. To this end, we manually curate raw data files, long and heterogeneous documentation, and expert-written publications from 17 publicly available U.S. national surveys, from which we extract 505 analytical queries grounded in real analytical practice. Solving these queries requires agents to first retrieve and integrate key information from multiple unstructured documents, before performing multi-step computations and writing executable code, which remains challenging for existing data analysis agents. To support the systematic evaluation under this setting, we develop LongTA, a tool-augmented agent framework that enables document access, retrieval, and code execution, and evaluate a range of proprietary and open-source models. Our experiments reveal substantial performance gaps even among state-of-the-art models, highlighting the challenges researchers should consider before applying LLM agents for decision support in real-world, high-stakes analytical settings.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。