评测大模型在无类型Python代码库中的类型推断能力，发现其一致性差。
TypyBench: Evaluating LLM Type Inference for Untyped Python Repositories
- 构建跨整个代码库的类型推断评测基准，含语义相似度与一致性指标
- 大模型在简单类型上表现尚可，但对嵌套复杂类型和全局一致性错误频发
- 揭示当前研究需从提升类型相似度转向保障代码库级类型一致
动态语言如Python的类型推断是软件工程中的长期挑战。尽管大语言模型（LLMs）在代码理解方面展现出潜力，但其类型推断能力仍缺乏深入探索。本文提出TypyBench，一个用于评估LLMs在完整Python代码库中类型推断性能的基准。该基准包含两项新指标：TypeSim，用于捕捉预测类型与真实类型间的细微语义关系；TypeCheck，用于评估代码库内的类型一致性。我们在50个高质量Python代码库构成的精选数据集上评估了多种LLMs，结果表明，尽管模型在TypeSim上表现良好，但在处理复杂嵌套类型时表现不佳，并存在显著的类型一致性错误。这些发现提示未来研究应从提升类型相似性转向解决代码库级别的类型一致性问题。TypyBench为这一新方向奠定了基础，提供了不同类型复杂度和使用场景下模型表现的洞察。代码与数据已开源于https://github.com/typybench/typybench。
原文摘要 · Abstract (English)
Type inference for dynamic languages like Python is a persistent challenge in software engineering. While large language models (LLMs) have shown promise in code understanding, their type inference capabilities remain underexplored. We introduce TypyBench, a benchmark designed to evaluate LLMs' type inference across entire Python repositories. TypyBench features two novel metrics: TypeSim, which captures nuanced semantic relationships between predicted and ground truth types, and TypeCheck, which assesses type consistency across codebases. Our evaluation of various LLMs on a curated dataset of 50 high-quality Python repositories reveals that, although LLMs achieve decent TypeSim scores, they struggle with complex nested types and exhibit significant type consistency errors. These findings suggest that future research should shift focus from improving type similarity to addressing repository-level consistency. TypyBench provides a foundation for this new direction, offering insights into model performance across different type complexities and usage contexts. Our code and data are available at https://github.com/typybench/typybench.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。