对比两种模型在六大数据集上的漏洞修复能力，发现通用性仍存挑战。
A Multi-Dataset Evaluation of Models for Automated Vulnerability Repair
- 用CodeBERT和CodeT5模型在四语言六数据集上测试漏洞修复效果
- 细调后模型在已见数据表现好，但对未知漏洞泛化能力差
- 代码上下文碎片化时CodeBERT更稳，复杂模式下CodeT5更强
软件漏洞带来重大安全威胁，需有效缓解。尽管自动化程序修复（APR）在一般性缺陷修复方面取得进展，但作为安全关键任务的漏洞修补仍研究不足。本研究评估预训练语言模型CodeBERT与CodeT5在六大数据集及四种编程语言上的自动漏洞修复表现，考察其准确率与对未知漏洞的泛化能力。结果表明，当上下文碎片化或稀疏时，CodeBERT表现相对更优；而CodeT5在捕捉复杂漏洞模式方面更具优势，并展现出更好的可扩展性。我们进一步在分布内（训练数据）与分布外（未见数据）数据集上测试微调模型。微调虽提升了分布内性能，但模型在未知数据上仍难以泛化，凸显鲁棒漏洞检测的挑战。本研究为模型性能提供基准，揭示泛化局限，并为实际安全应用中的自动化漏洞修复提供可操作洞见。
原文摘要 · Abstract (English)
Software vulnerabilities pose significant security threats, requiring effective mitigation. While Automated Program Repair (APR) has advanced in fixing general bugs, vulnerability patching, a security-critical aspect of APR remains underexplored. This study investigates pre-trained language models, CodeBERT and CodeT5, for automated vulnerability patching across six datasets and four languages. We evaluate their accuracy and generalization to unknown vulnerabilities. Results show that while both models face challenges with fragmented or sparse context, CodeBERT performs comparatively better in such scenarios, whereas CodeT5 excels in capturing complex vulnerability patterns. CodeT5 also demonstrates superior scalability. Furthermore, we test fine-tuned models on both in-distribution (trained) and out-of-distribution (unseen) datasets. While fine-tuning improves in-distribution performance, models struggle to generalize to unseen data, highlighting challenges in robust vulnerability detection. This study benchmarks model performance, identifies limitations in generalization, and provides actionable insights to advance automated vulnerability patching for real-world security applications.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。