arXiv:2504.10557cs.SEcs.AI2025-04被引 13

用代码混淆测试大模型真理解,发现通用模型反而更抗干扰。

The Code Barrier: What LLMs Actually Understand?

  • 通过控制变量的代码混淆测试模型语义理解能力。
  • 混淆越复杂,模型表现越差,通用模型比专用模型更稳定。
  • 适合关注代码安全、逆向工程和对抗性分析的研究者。

代码理解是自动化软件开发的核心能力。尽管大语言模型在诸多软件工程任务中表现优异,但其是否具备超越简单词元识别的真实语义理解仍不明确。本研究采用代码混淆作为结构化测试框架,系统性地对源代码施加可控的混淆操作,并通过生成准确描述和执行去混淆两项互补任务评估模型理解能力。实验涵盖13个前沿模型,包括StarCoder2等代码专用模型与GPT-4o等通用模型,基于CodeNet构建的250个过滤后的Java编程问题及其解答进行评估。结果表明,随着混淆复杂度增加,模型性能显著下降;出人意料的是,通用模型表现出更强的鲁棒性,优于代码专用模型。尽管部分模型能识别混淆技术,但重构底层程序逻辑的能力依然受限,揭示其语义表征机制存在局限。该研究提出一种新的代码理解评估方法,建立了面向安全关键场景(如逆向工程、对抗性代码分析)的基准体系。

原文摘要 · Abstract (English)

Understanding code represents a core ability needed for automating software development tasks. While foundation models like LLMs show impressive results across many software engineering challenges, the extent of their true semantic understanding beyond simple token recognition remains unclear. This research uses code obfuscation as a structured testing framework to evaluate LLMs' semantic understanding capabilities. We methodically apply controlled obfuscation changes to source code and measure comprehension through two complementary tasks: generating accurate descriptions of obfuscated code and performing deobfuscation, a skill with important implications for reverse engineering applications. Our testing approach includes 13 cutting-edge models, covering both code-specialized (e.g., StarCoder2) and general-purpose (e.g., GPT-4o) architectures, evaluated on a benchmark created from CodeNet and consisting of filtered 250 Java programming problems and their solutions. Findings show a statistically significant performance decline as obfuscation complexity increases, with unexpected resilience shown by general-purpose models compared to their code-focused counterparts. While some models successfully identify obfuscation techniques, their ability to reconstruct the underlying program logic remains constrained, suggesting limitations in their semantic representation mechanisms. This research introduces a new evaluation approach for assessing code comprehension in language models and establishes empirical baselines for advancing research in security-critical code analysis applications such as reverse engineering and adversarial code analysis.

代码理解大模型评估安全分析代码混淆

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。