用大模型预测安全漏洞报告，提示工程更敏感但误报高，微调模型更准但漏报多。
Evaluating Large Language Models for Security Bug Report Prediction
- 对比提示工程与微调两种大模型方法在漏洞报告预测中的表现。
- 提示模型召回率达74%但精度仅22%，微调模型精度达75%但召回率仅36%。
- 微调模型推理速度比提示模型快50倍，适合对效率要求高的场景。
早期发现安全漏洞报告（SBRs）对于及时缓解漏洞至关重要。本文评估了基于提示工程和微调的大语言模型（LLMs）在预测SBRs中的表现。结果显示两种方法存在明显权衡：使用提示的专有模型在所有数据集上平均达到77%的G-measure和74%的召回率，但误报率较高，平均精确率仅为22%；而微调模型则表现出相反特征，整体G-measure为51%，精确率达到75%，但召回率下降至36%。尽管微调模型需要一次性构建投入，其在最大数据集上的推理速度比专有模型快至多50倍。这些发现表明，进一步探索大模型在SBR预测中的潜力仍有必要。
原文摘要 · Abstract (English)
Early detection of security bug reports (SBRs) is critical for timely vulnerability mitigation. We present an evaluation of prompt-based engineering and fine-tuning approaches for predicting SBRs using Large Language Models (LLMs). Our findings reveal a distinct trade-off between the two approaches. Prompted proprietary models demonstrate the highest sensitivity to SBRs, achieving a G-measure of 77% and a recall of 74% on average across all the datasets, albeit at the cost of a higher false-positive rate, resulting in an average precision of only 22%. Fine-tuned models, by contrast, exhibit the opposite behavior, attaining a lower overall G-measure of 51% but substantially higher precision of 75% at the cost of reduced recall of 36%. Though a one-time investment in building fine-tuned models is necessary, the inference on the largest dataset is up to 50 times faster than that of proprietary models. These findings suggest that further investigations to harness the power of LLMs for SBR prediction are necessary.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。