用语法语义双管齐下的方法,精准解析代码笔记本的数据流动。
CRABS: A syntactic-semantic pincer strategy for bounding LLM interpretation of Python notebooks
- 结合浅层语法分析与大模型零样本学习,定位细胞间数据输入输出。
- 在50个高赞Kaggle笔记本上,98%的歧义被正确解决,信息流识别F1达98%。
- 适合需要理解复杂代码笔记本但无法重跑的人,如研究人员和开发者。
识别数据科学与机器学习Python笔记本中的信息流与操作,对评估、复用和适应新任务至关重要。由于数据与软件依赖难以解决,通过重运行分析笔记本往往不可行。尽管预训练于大规模代码库的大语言模型(LLM)在无需执行的情况下理解代码方面表现出色,但我们发现其在真实笔记本中因幻觉和长上下文问题而失效。为此,我们提出一个笔记本理解任务,生成信息流图与单元格执行依赖图,并展示一种使用有限语法分析辅助完整理解的钳形策略。我们的捕获与化解辅助边界策略(CRABS)利用浅层语法解析和抽象语法树(AST)分析,捕捉细胞间输入输出集的上下界估计,再通过大模型逐单元格零样本学习解决剩余歧义,从而确定每个单元的真实数据输入输出。我们在包含50个代表性高赞Kaggle笔记本的标注数据集上评估,共涵盖3454个实际细胞输入输出。大模型成功解决了1397/1425(98%)由语法结构分析遗留的歧义。在50个笔记本上,CRABS平均实现98%的细胞间信息流识别率与99%的传递执行依赖识别率。
原文摘要 · Abstract (English)
Recognizing the information flows and operations comprising data science and machine learning Python notebooks is critical for evaluating, reusing, and adapting notebooks for new tasks. Investigating a notebook via re-execution often is impractical due to the challenges of resolving data and software dependencies. While Large Language Models (LLMs) pre-trained on large codebases have demonstrated effectiveness in understanding code without running it, we observe that they fail to understand some realistic notebooks due to hallucinations and long-context challenges. To address these issues, we propose a notebook understanding task yielding an information flow graph and corresponding cell execution dependency graph for a notebook, and demonstrate the effectiveness of a pincer strategy that uses limited syntactic analysis to assist full comprehension of the notebook using an LLM. Our Capture and Resolve Assisted Bounding Strategy (CRABS) employs shallow syntactic parsing and analysis of the abstract syntax tree (AST) to capture the correct interpretation of a notebook between lower and upper estimates of the inter-cell I/O set$\unicode{x2014}$the flows of information into or out of cells via variables$\unicode{x2014}$then uses an LLM to resolve remaining ambiguities via cell-by-cell zero-shot learning, thereby identifying the true data inputs and outputs of each cell. We evaluate and demonstrate the effectiveness of our approach using an annotated dataset of 50 representative, highly up-voted Kaggle notebooks that together represent 3454 actual cell inputs and outputs. The LLM correctly resolves 1397 of 1425 (98%) ambiguities left by analyzing the syntactic structure of these notebooks. Across 50 notebooks, CRABS achieves average F1 scores of 98% identifying cell-to-cell information flows and 99% identifying transitive cell execution dependencies.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。