首个跨版本跨仓库的漏洞补丁回滚基准测试,揭示现有工具真实表现差距。
Benchmarking Automated Security Patch Backporting: How Far Are We?

- 构建1234个跨场景补丁回滚案例与统一评估框架
- 复杂补丁成功率从85.2%降至24.0%,工具性能显著下降
- 发现四大失效根源,强调可执行验证比静态匹配更关键
自动化安全补丁回滚对缓解N日漏洞至关重要。近期工具在各自数据集上报告成功率超80%,但评估多限于同源环境。本文提出Porting Benchmark,包含1,234个跨版本、跨分支、跨仓库的补丁回滚案例,并提供统一评估框架。在该基准下,评估五种工具(基于程序分析、LLM提示、LLM代理)。结果表明:统一评估改变性能格局——PortGPT和TSBPort在复现数据集仍较强,而FixMorph和Mystique性能大幅下降。结构复杂补丁成功率急剧下滑:提交级最佳成功率由类型I的85.2%降至类型IV的24.0%。识别出四大根因:目标API感知缺失、跨版本语义不一致、非局部依赖传播失败、补丁构造或定位失败。进一步在45个动态验证子集上发现:参考匹配分数无法完全反映实际修复效果——精确匹配低估困难目标适配,可执行验证揭示静态匹配忽略的集成缺陷。可执行反馈微调对最难案例有有限但可观的恢复效果。
原文摘要 · Abstract (English)
Automated security patch backporting is critical for mitigating N-day vulnerabilities. Recent tools report success rates above 80% on their respective datasets. However, these evaluations are often confined to homogeneous environments, such as one repository or specific project versions. Consequently, it remains unclear how well these tools generalize beyond their originally targeted scenarios. We present Porting Benchmark, a curated dataset of 1,234 security patch backporting cases spanning cross-version, cross-branch, and cross-repository scenarios, paired with a common evaluation framework. Using this benchmark, we evaluate five tools spanning program analysis, LLM prompting, and LLM agents under aligned settings. Our results show that aligned evaluation changes the apparent performance landscape: PortGPT and TSBPort remain comparatively strong on the Replication Dataset, while FixMorph and Mystique degrade substantially under the common protocol. Performance degrades sharply on structurally complex patches: the best commit-level success rate falls from 85.2% on Type-I patches to 24.0% on Type-IV. We identify four root-cause categories (missing target API awareness, cross-version semantic mismatch, non-local dependency propagation failure, and patch construction or localization failure) and derive concrete directions for next-generation tool design. On a 45-case dynamically validated subset with verified test cases and constructed POCs, we further observe that reference-based benchmark scores do not fully capture real-world remediation: exact match sharply under-credits harder target adaptations, while executable validation reveals residual integration failures in the target that static reference agreement misses. Executable-feedback refinement provides limited but measurable recovery on the hardest executable cases.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。