为安全漏洞检测的LLM设计多角色评估框架，让不同负责人看得见真实表现差异。
Seclens: Role-specific Evaluation of LLM's for security vulnerablity detection
- 按5类角色定制权重，用12-16个维度生成决策得分
- 同一模型在不同角色下得分差达31分，如Qwen3-Coder从A到D
- 适用于企业安全、研发与AI管理决策者参考
现有基于LLM的漏洞检测基准将模型性能压缩为单一指标，无法反映不同利益相关方的差异化关注点。例如，首席信息安全官（CISO）更关注关键漏洞的高召回率，工程负责人重视降低误报，而人工智能负责人则需平衡能力与成本。为此，我们提出SecLens-R，一个围绕35个共享维度、分为7个测量类别的多利益相关方评估框架。该框架定义了五种角色特定加权配置：CISO、首席人工智能官、安全研究员、工程负责人及AI行为体。每种配置选取12至16个维度，权重总和为80，生成0到100之间的综合决策得分。我们在涵盖10种编程语言和8个符合OWASP标准的漏洞类别的406个任务数据集上，对12个前沿模型进行了评估，覆盖代码提示（CIP）与工具使用（TU）两种场景。结果表明，不同角色视角下评分差异显著，同一模型得分跨度最高达31分。例如，Qwen3-Coder在工程负责人配置下得分为76.3（A），但在CISO配置下仅为45.2（D），GPT-5.4亦呈现类似差距。这些发现表明，漏洞检测本质上是多目标问题，而利益相关方感知的评估能揭示单一聚合指标所掩盖的深层洞察。
原文摘要 · Abstract (English)
Existing benchmarks for LLM-based vulnerability detection compress model performance into a single metric, which fails to reflect the distinct priorities of different stakeholders. For example, a CISO may emphasize high recall of critical vulnerabilities, an engineering leader may prioritize minimizing false positives, and an AI officer may balance capability against cost. To address this limitation, we introduce SecLens-R, a multi-stakeholder evaluation framework structured around 35 shared dimensions grouped into 7 measurement categories. The framework defines five role-specific weighting profiles: CISO, Chief AI Officer, Security Researcher, Head of Engineering, and AI-as-Actor. Each profile selects 12 to 16 dimensions with weights summing to 80, yielding a composite Decision Score between 0 and 100. We apply SecLens-R to evaluate 12 frontier models on a dataset of 406 tasks derived from 93 open-source projects, covering 10 programming languages and 8 OWASP-aligned vulnerability categories. Evaluations are conducted across two settings: Code-in-Prompt (CIP) and Tool-Use (TU). Results show substantial variation across stakeholder perspectives, with Decision Scores differing by as much as 31 points for the same model. For instance, Qwen3-Coder achieves an A (76.3) under the Head of Engineering profile but a D (45.2) under the CISO profile, while GPT-5.4 shows a similar disparity. These findings demonstrate that vulnerability detection is inherently a multi-objective problem and that stakeholder-aware evaluation provides insights that single aggregated metrics obscure.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。