用大模型自动评估逆向工程输出,无需参考代码。
Beyond Text Matching: Towards Reference-Free Evaluation for Human-Oriented Binary Reverse Engineering

- 采用大模型做裁判,实现无参考的自动化评估。
- 在三项任务中相关性达63.2%,超越传统方法。
- 自适应选择最佳评估配置,省成本且更准确。
面向人类的二进制逆向工程(HOBRE)旨在将反编译伪代码转化为更易理解的形式,降低分析认知负担并提升效率。然而,可靠评估HOBRE输出仍是根本挑战:人工评估成本高、难以扩展;现有自动化指标或需可执行测试用例和运行环境(常不可得),或依赖高质量源代码参考(通常无法获取),无法捕捉语义等价但词法多样的输出。尽管大模型作为裁判(LLM-as-a-Judge)天然适合此类评估,其有效性仍待探索。本文首次系统研究了该范式在三个代表性任务中的应用:函数名恢复、二进制代码摘要生成与反编译优化。我们提出BinJudgeBench,首个基于多维人类判断的专家标注、无参考评估基准,其中大模型裁判平均相关性达63.20%,优于传统自动化指标的35.04%。通过分析不同骨干模型、提示策略与解码温度下的裁判表现,发现不存在通用最优配置,最优设置随任务与样本变化。为此,我们提出BinJudge,采用轻量级路由机制,为每项任务与样本动态选择最优裁判配置。该方法使与人类专家的相关性提升4.5%-24.7%,并降低API成本至静态最优配置的0.06×-0.84×,提供了一种可扩展、低成本、高保真的自动化评估方案。
原文摘要 · Abstract (English)
Human-Oriented Binary Reverse Engineering (HOBRE) aims to transform decompiled pseudocode into a more human-friendly representation, thereby reducing the cognitive burden of reverse analysis and improving efficiency. However, reliably evaluating HOBRE outputs remains a fundamental challenge: human evaluation is costly, time-consuming, and difficult to scale, while existing automated metrics either require executable test cases and runtime environments that are often unavailable for real-world binaries, or rely on high-quality source code references that are typically inaccessible and fail to capture semantically equivalent but lexically diverse outputs. Although LLM-as-a-Judge paradigm is naturally well-suited to HOBRE evaluation, its effectiveness remains underexplored. This paper presents the first systematic investigation of the LLM-as-a-Judge paradigm for HOBRE across three representative tasks: function name recovery, binary code summarization, and decompilation optimization. We introduce BinJudgeBench, the first expert-annotated, reference-free evaluation benchmark based on multi-dimensional human judgment, where LLM-as-a-Judge achieves an average correlation of 63.20\% with human judgment, outperforming traditional automated metrics at 35.04\%. By analyzing judge configurations across backbone LLMs, prompting strategies, and decoding temperatures, we find that no ``one-size-fits-all'' configuration exists, as the optimal setup varies across tasks and individual samples. To address this, we propose BinJudge, which employs a lightweight routing mechanism to adaptively select the optimal judge configuration for each task and sample. BinJudge improves correlation with human experts by 4.5\%-24.7\% and reduces API cost to 0.06$\times$-0.84$\times$ of that of static best configurations, providing a scalable, cost-effective, and high-fidelity automated evaluation scheme for HOBRE.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。