用笔记和搜索增强大模型数据分析能力,让其更懂复杂任务推理。
Jupiter: Enhancing LLM Data Analysis Capabilities via Notebook and Inference-Time Value-Guided Search
- 从真实代码笔记中提取带工具调用的多步分析任务与解法。
- 在InfiAgent-DABench上,14B模型解决率达86.38%,媲美GPT-4o。
- 适合研究大模型推理、自动化数据分析或智能编程的人参考。
大型语言模型在自动化数据科学流程方面展现出巨大潜力,但现有模型在多步推理和工具使用方面仍存在不足,限制了其在复杂数据分析任务中的表现。为此,我们提出一个可扩展的流水线,从真实的Jupyter笔记和相关数据文件中提取高质量、基于工具的数据分析任务及其可执行的多步解决方案。基于该流水线,我们构建了NbQA——一个大规模的标准任务-解法对数据集,反映真实数据科学场景中的工具使用模式。为进一步提升多步推理能力,我们提出了Jupiter框架,将数据分析建模为搜索问题,采用蒙特卡洛树搜索(MCTS)生成多样化的解题路径以训练价值模型。推理时,Jupiter结合价值模型与节点访问次数,高效生成可执行的多步计划,仅需最少搜索步骤。实验结果表明,Qwen2.5-7B和14B-Instruct模型在NbQA上的任务解决率分别为77.82%和86.38%,达到或超过GPT-4o及先进代理框架。进一步评估显示,其在多样化多步推理任务中具备更强的泛化能力和工具使用推理能力。代码与数据见https://github.com/microsoft/Jupiter。
原文摘要 · Abstract (English)
Large language models (LLMs) have shown great promise in automating data science workflows, but existing models still struggle with multi-step reasoning and tool use, which limits their effectiveness on complex data analysis tasks. To address this, we propose a scalable pipeline that extracts high-quality, tool-based data analysis tasks and their executable multi-step solutions from real-world Jupyter notebooks and associated data files. Using this pipeline, we introduce NbQA, a large-scale dataset of standardized task-solution pairs that reflect authentic tool-use patterns in practical data science scenarios. To further enhance multi-step reasoning, we present Jupiter, a framework that formulates data analysis as a search problem and applies Monte Carlo Tree Search (MCTS) to generate diverse solution trajectories for value model learning. During inference, Jupiter combines the value model and node visit counts to efficiently collect executable multi-step plans with minimal search steps. Experimental results show that Qwen2.5-7B and 14B-Instruct models on NbQA solve 77.82% and 86.38% of tasks on InfiAgent-DABench, respectively-matching or surpassing GPT-4o and advanced agent frameworks. Further evaluations demonstrate improved generalization and stronger tool-use reasoning across diverse multi-step reasoning tasks. Code and data are available at https://github.com/microsoft/Jupiter.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。