首个支持真实划分的Java漏洞检测基准，统一评估多类模型。
JavaVulBench: A Java Vulnerability Benchmark with Realistic Splits, a Unified Multi-Backend Harness, and a Leakage-Aware Evaluation Mode
- 构建含3万+方法的漏洞数据集，支持五种真实划分策略。
- 统一接口评估12个模型，涵盖编码器与大模型等多种架构。
- 内置污染审计机制，确保测试集未被模型提前记忆。
我们发布 extsc{JavaVulBench}，一个用于 Java 漏洞检测的基准数据集与评估工具。该数据集包含约 30,600 个 Java 方法，覆盖 1,740 个 CVE 和 700 多个项目，标注粒度达方法级和行级，并提供每个 CVE 的发布日期。支持五种现实划分策略：随机、项目不交集、时间序列、去重、未见 CWE 类型。评估框架统一支持三种后端：编码器分类器、本地生成模型（通过 Ollama 服务）、以及通过 OpenRouter 路由的 API 服务大模型。十二个参考检测器（包括 CodeBERT、GraphCodeBERT、UniXcoder、DeepSeek-Coder-1.3B，以及八款 API/开源大模型如 GPT-4o、Claude Sonnet 4、Qwen-2.5-Coder-14B 等）可在同一命令下进行公平评估。所有模型均附带预训练污染审计，帮助用户区分真正未见过的测试 CVE 与可能被记忆的样本。数据、代码及微调检查点已存档于 Zenodo [31]，演示视频见 YouTube (https://www.youtube.com/watch?v=nMTX_hqkuoM)。
原文摘要 · Abstract (English)
We release \textsc{JavaVulBench}, a benchmark dataset and evaluation harness for Java vulnerability detection. The dataset contains $\sim$30{,}600 Java methods spanning 1{,}740 CVEs and 700+ projects, labelled at both method and line granularity, with per-CVE publication dates and five realistic split strategies: random, project-disjoint, temporal, deduplicated, and unseen CWE-family. The harness provides a single \texttt{LlmPrediction} schema across three backend families (encoder classifiers, local generative models served by Ollama, and API-served LLMs routed through OpenRouter) so that twelve reference detectors CodeBERT, GraphCodeBERT, UniXcoder, DeepSeek-Coder-1.3B, and eight API/open-weight LLMs (GPT-4o, GPT-4.1-mini, Claude Sonnet~4, DeepSeek-v3, DeepSeek-Coder-v2, Qwen-2.5-Coder-14B/7B, CodeLlama-13B) are evaluated under identical conditions from a single command. A pre-training contamination audit is shipped alongside every model so users can separate genuinely unseen test CVEs from potentially memorised ones. Data, code, and fine-tuned checkpoints are archived on Zenodo [31] and short demonstration video is available on YouTube (https://www.youtube.com/watch?v=nMTX\_hqkuoM) https://www.youtube.com/watch?v=nMTX_hqkuoM.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。