arXiv:2505.11340cs.SEcs.AI2025-05ACL被引 13

首个面向真实场景的反编译器评估基准,验证LLM在可读性上的优势。

DecompileBench: A Comprehensive Benchmark for Evaluating Decompilers in Real-World Scenarios

  • 构建23,400个真实程序函数库,结合运行时验证与LLM评分
  • LLM反编译器代码可读性胜过商用工具,正确率低52.2%
  • 适合安全研究人员选型与反编译技术研究者参考

反编译器是漏洞发现和恶意软件分析等关键安全任务的基础工具,但现有评估方法主要依赖合成微基准或主观人工评分,难以衡量真实场景中的语义保真度与分析师可用性。本文提出DecompileBench,首个综合性评估框架,通过三个核心组件实现对反编译器在逆向工程流程中的有效评估:真实世界函数提取(来自130个实际程序的23,400个函数)、运行时感知验证,以及基于LLM-as-Judge的自动化人本评估。通过对六款工业级反编译器与六种新兴的LLM驱动方法进行系统对比,发现尽管LLM方法的功能正确率低52.2%,但在代码可读性方面显著优于商业工具。该结果凸显了基于LLM的方法在以人为核心的逆向工程中的潜力。我们开源DecompileBench,为推进反编译器研究及辅助安全专家根据需求选择合适工具提供支持。

原文摘要 · Abstract (English)

Decompilers are fundamental tools for critical security tasks, from vulnerability discovery to malware analysis, yet their evaluation remains fragmented. Existing approaches primarily focus on syntactic correctness through synthetic micro-benchmarks or subjective human ratings, failing to address real-world requirements for semantic fidelity and analyst usability. We present DecompileBench, the first comprehensive framework that enables effective evaluation of decompilers in reverse engineering workflows through three key components: \textit{real-world function extraction} (comprising 23,400 functions from 130 real-world programs), \textit{runtime-aware validation}, and \textit{automated human-centric assessment} using LLM-as-Judge to quantify the effectiveness of decompilers in reverse engineering workflows. Through a systematic comparison between six industrial-strength decompilers and six recent LLM-powered approaches, we demonstrate that LLM-based methods surpass commercial tools in code understandability despite 52.2% lower functionality correctness. These findings highlight the potential of LLM-based approaches to transform human-centric reverse engineering. We open source \href{https://github.com/Jennieett/DecompileBench}{DecompileBench} to provide a framework to advance research on decompilers and assist security experts in making informed tool selections based on their specific requirements.

反编译LLM评估安全分析

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。