提出新方法Reforge,精准评估大模型在反编译函数命名中的真实能力
REFORGE: A Method for Benchmarking LLMs' Reverse Engineering Capabilities in Decompiled Binary Function Naming
- 构建从源码到反编译函数的可追溯验证流程,解决地面真值生成难题
- 发现优化级别提升使高置信度可用函数率从87.2%降至65.9%
- 揭示现有评估的幸存者偏差问题,倡导带不确定性的评测范式
大型语言模型(LLMs)正被越来越多地应用于逆向工程任务,近期威胁情报报告表明其已融入实际攻防工作流。然而,关于其能力的宣称远超我们衡量的能力。现有针对LLM辅助二进制分析的基准测试将函数级真值构建视为已解决的预处理步骤,且未披露有多少函数可被可靠评估。本文指出,公平评估的主要障碍并非模型能力,而是编译优化下二进制与源码对齐的可靠性问题。本文提出Reforge,一个具备溯源追踪的流水线,通过编译、DWARF提取、语法分析、对齐和反编译,构建函数级真值,并将对齐不确定性建模为八门控置信度漏斗与三层分层结构。在受控微基准测试中,高置信度可用率在优化级别间从87.2%降至65.9%,未配对比较因幸存者偏差而夸大了优化带来的性能衰减。对七种主流LLM在函数命名上的概念验证评估,证实了该方法的有效性,普遍支持采用不确定性感知的基准评测实践。
原文摘要 · Abstract (English)
Large language models (LLMs) are increasingly applied to reverse-engineering tasks, and recent threat-intelligence reporting shows them operating inside live offensive-security workflows. Claims about their capability, however, outpace our ability to measure it. Existing benchmarks for LLM-assisted binary analysis treat the construction of function-level ground truth as a solved pre-processing step and report accuracy without disclosing how many functions were reliably evaluable. We argue that the principal obstacle to fair evaluation is not model capability but the reliability of binary-to-source alignment under compiler optimization. This paper presents Reforge, a provenance-tracked pipeline that constructs function-level ground truth from C source through compilation, DWARF and syntactic extraction, alignment, and decompilation, and that operationalizes alignment uncertainty as an eight-gate confidence funnel with three-tier stratification. On a controlled micro-benchmark, high-confidence yield falls from 87.2% to 65.9% across optimization levels, and unpaired comparisons overstate optimization-induced performance decay through survivorship bias. A proof-of-concept evaluation of seven contemporary LLMs on function naming demonstrates the validity of the concept and generally motivates an uncertainty-aware benchmarking practice.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。