新基准测试发现大模型做长期数据分析时极易失忆，准确率最高仅48%。
LongDS-Bench: On the Failure of Long-Horizon Agentic Data Analysis
- 设计68个真实数据任务，要求模型持续追踪分析状态变化
- 平均11.3轮依赖跨度，后期准确率比初期下降近47个百分点
- 适合研究长期推理、多轮交互的AI系统开发者
现实中的数据分析具有高度迭代性，但现有基准大多只评估孤立或短周期任务，无法检验智能体在长周期中对分析上下文的持续追踪能力。我们提出LongDS基准，涵盖68个来自真实Kaggle笔记本的任务，覆盖地理科学、商业、教育等六个领域，共2,225轮对话，平均依赖跨度为11.3轮。任务围绕状态演化模式（如反事实扰动、回滚、多状态组合）设计。评估五个顶尖模型后发现，最优模型平均准确率仅48.45%，从早期到后期性能下降近47分，长周期错误占失败案例的52%至69%。进一步分析表明，增加交互步数并不提升性能，说明关键瓶颈在于维持正确分析状态而非扩大交互预算。代码与数据已开源：https://github.com/zjunlp/DataMind。
原文摘要 · Abstract (English)
Real-world data analysis is inherently iterative, yet existing benchmarks mostly evaluate isolated or short interactive tasks, leaving agents' ability to track evolving analytical context over long horizons untested. We introduce LongDS, a benchmark for long-horizon, multi-turn data analysis where agents must maintain, update, restore, and compose evolving analytical states. LongDS comprises 68 tasks constructed from real-world Kaggle notebooks, spanning 2,225 turns across six domains including Geoscience, Business, and Education. Tasks are designed around state-evolution patterns (e.g., counterfactual perturbation, rollback, multi-state composition), with an average dependency span of 11.3 turns. Evaluating five state-of-the-art models, we find that the best model reaches only 48.45% average accuracy, performance drops nearly 47 points from early to late turns, and long-horizon errors account for 52%--69% of failures. Further analysis shows that additional agent steps do not necessarily improve performance, suggesting that the key bottleneck is maintaining a correct analytical state rather than increasing interaction budget. We release LongDS to support research on reliable long-horizon agentic data analysis. Code and data are released at https://github.com/zjunlp/DataMind.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。