测试不同代码长度对10个大模型漏洞检测能力的影响
Evaluating Large Language Models in Vulnerability Detection Under Variable Context Windows
- 用分词后代码长度变化测试模型表现
- 部分模型如GPT-4对输入长度不敏感，部分显著下降
- 建议优化模型对输入长度的鲁棒性及代码预处理
本研究考察了分词后Java代码长度对十种主流大语言模型在漏洞检测中准确率与明确性的影响。基于卡方检验和已知真实标签，发现模型间表现不一致：GPT-4、Mistral和Mixtral等模型表现出鲁棒性，而其他模型则显示出分词长度与性能之间存在显著关联。研究建议未来大模型开发应着重降低输入长度对检测效果的影响。此外，能够减少分词数量同时保持代码结构的预处理技术，有望提升大模型在漏洞检测任务中的准确性和明确性。
原文摘要 · Abstract (English)
This study examines the impact of tokenized Java code length on the accuracy and explicitness of ten major LLMs in vulnerability detection. Using chi-square tests and known ground truth, we found inconsistencies across models: some, like GPT-4, Mistral, and Mixtral, showed robustness, while others exhibited a significant link between tokenized length and performance. We recommend future LLM development focus on minimizing the influence of input length for better vulnerability detection. Additionally, preprocessing techniques that reduce token count while preserving code structure could enhance LLM accuracy and explicitness in these tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。