为大模型调用与代码间的语义断层构建可分析的映射模型
Reachability Across the NL/PL Boundary: A Taxonomy-Driven Dataflow Model for LLM-Integrated Applications
- 用25类标签刻画提示词占位符到输出的可达关系
- 在真实漏洞检测中实现F1=100%,比基线提升近一倍
- 适合安全分析、代码审计人员使用,尤其关注大模型集成应用
大模型API已成为标准编程单元,但其与代码间形成自然语言与程序语言的边界,破坏传统数据流分析。运行时值通过模板占位符注入大模型,经黑盒转换后以代码、JSON或文本形式返回,下游逻辑消耗该结果。现有污点分析和程序切片依赖调用方对输入输出的明确定义,而大模型调用缺乏此类数据流摘要,导致分析失效。本文提出PRISM,首个针对该边界的可达性模型。它将大模型调用的缺失数据流摘要抽象为占位符到输出的可达性,并基于量化信息流理论建立包含25个标签的有限分类体系,覆盖信息保留程度与输出模态两个维度。每个标签生成一个占位符的可达谓词。模型在标注上具有一致性(Fleiss' kappa ≥ 0.72),覆盖8,119个真实样本对,无不可分类项,发现概率为0.09%。在污点分析中,其准确率接近基线两倍,达F1=81.7%;在六个真实OpenClaw漏洞案例中,成功检测所有漏洞路径并验证所有修复(F1=100%)。在反向切片中,能剔除约四分之一无关代码而不丢失真实依赖。
原文摘要 · Abstract (English)
LLM API calls have become a standard programming primitive, but they create a program boundary that disrupts traditional dataflow analysis. A runtime value may be inserted into a natural-language prompt through a template placeholder, transformed opaquely by the LLM, and returned as code, JSON, or text consumed by downstream logic. Existing analyses such as taint analysis and program slicing require a dataflow summary that describes how a callee maps inputs to outputs; an LLM call provides no such summary, breaking analysis at what we call the NL/PL boundary. We introduce PRISM, the first reachability model for this boundary. PRISM abstracts the missing dataflow summary of an LLM call as placeholder-to-output reachability. Because the LLM's internal transformation is opaque, the only observable signal is the input-output relationship, which spans an unbounded range of behaviors. PRISM therefore uses a finite taxonomy grounded in quantitative information flow theory. It classifies placeholder-output behavior into 25 labels along two dimensions: information preservation and output modality. Each label yields a reachability predicate for a placeholder. The model is sound with respect to its labeling, with residual error bounded empirically. PRISM is dependable and effective. Independent models and human annotators assign its labels consistently (Fleiss' kappa >= 0.72), and the labels cover 8,119 real-world pairs, leaving no pair unclassifiable; the Good-Turing discovery probability is 0.09%. For taint analysis, PRISM nearly doubles the conservative baseline and outperforms a direct LLM baseline, achieving F1 = 81.7%. Across six real OpenClaw CVEs, it detects every vulnerable flow and confirms every patch (F1 = 100%). In backward slicing, it removes about a quarter of irrelevant code without discarding any true dependency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。