LLM生成代码常带高危依赖版本,存在普遍安全风险。
Correct Code, Vulnerable Dependencies: A Large Scale Measurement Study of LLM-Specified Library Versions

- 分析10个LLM在1000个编程任务中指定的依赖版本
- 36.7%-55.7%的版本含已知漏洞,多数为严重级别
- 所有模型都倾向选少数高危版本,适合开发者与安全研究者关注
大型语言模型(LLMs)已深度参与软件开发流程,其生成的代码常包含带具体版本标识的第三方库(TPL)导入。这些版本选择可能带来安全和兼容性风险,但尚未被系统研究。本研究首次对LLM生成的Python代码中的版本级风险进行大规模测量,评估10个LLM在PinTrace基准(1000个Stack Overflow编程任务)上的表现。直接提示时,模型指定版本的比例为26.83%-95.18%,而生成manifest文件时降至6.45%-59.19%。其中36.70%-55.70%的任务包含至少一个已知CVE,62.75%-74.51%的漏洞等级为关键或高危。72.27%-91.37%的漏洞在模型知识截止前已公开。所有模型均趋向于少数高危版本,表明系统性偏差而非孤立错误。静态兼容率仅19.70%-63.20%,安装失败为主要原因;动态测试通过率仅为6.49%-48.62%。进一步实验确认问题源于版本选择而非代码质量,且外部约束可显著降低漏洞暴露与兼容失败。研究揭示了LLM版本选择是此前被忽视的重要风险面。相关发现已通报各模型社区,多数组承认该问题。全部代码与数据集已在https://github.com/dw763j/PinTrace 开源。
原文摘要 · Abstract (English)
Large language models (LLMs) are now largely involved in software development workflows, and the code they generate routinely includes third-party library (TPL) imports annotated with specific version identifiers. These version choices can carry security and compatibility risks, yet they have not been systematically studied. We present the first large-scale measurement study of version-level risk in LLM-generated Python code, evaluating 10 LLMs on PinTrace, a curated benchmark of 1,000 Stack Overflow programming tasks. LLMs tend to specify version identifiers when directly prompted at 26.83%-95.18%, while down to 6.45%-59.19% in creating a manifest file directly. Among the specified versions, 36.70%-55.70% of tasks contain at least one known CVE, and 62.75%-74.51% of them carry Critical or High severity ratings. In 72.27%-91.37% of cases, the associated CVEs were publicly disclosed before the model's knowledge cutoff. The statistics show all models converge on the same small set of risky release versions, indicating a systemic bias rather than isolated model error. Static compatibility rates range from 19.70% to 63.20%, with installation failure as the dominant cause. The dynamic test cases confirm the pattern by 6.49%-48.62% pass rates. Further experiments confirm that these failures are attributable to version selection rather than code quality, and that externally anchored version constraints substantially reduce both vulnerability exposure and compatibility failures. Our findings reveal LLM version selection as a first-class, previously overlooked risk surface in LLM-based development. We disclosed these findings to the community of the evaluated models, and several confirmed the issue. All the code and dataset have been released for open science at https://github.com/dw763j/PinTrace.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。