评估神经去编译在Dart语言上的微调效果与指标有效性，发现现有方法存在显著局限。
Evaluating Fine-Tuning and Metrics for Neural Decompilation of Dart AOT Binaries
- 对比六种微调模型，验证不同参数规模下的性能变化
- 发现微调反而导致通过率下降5.65个百分点，且指标间存在矛盾
- 强调应以通过率（pass@k）为主要评估标准，适合代码生成研究者参考
神经去编译作为代码生成问题受到关注，但对现代语言的评估方法仍不成熟。本文系统研究了Dart提前编译（AOT）神经去编译中微调的有效性与指标合理性。在新构建的154任务HumanEval-Dart基准上，评估了三种基础架构（4B-8B参数）的六种微调模型变体，使用CodeBLEU、compile@k和pass@k三个指标。研究发现：第一，无微调配置带来统计显著的pass@k提升；最强基线模型Qwen3-8B微调后导致-5.65个百分点显著下降（p<0.001）；第二，4B模型受Swift训练干扰显著（-2.66 pp, p<0.001），而8B模型则不显著，符合缩放假设；第三，CodeBLEU与compile@k可提升，但pass@k反向下降，说明表面相似性误导评估。错误分析显示，汇编指令长度是任务难度最强预测因子（p=0.001），200条为能力断崖点。我们贡献了HumanEval-Dart基准、Dart适配的CodeBLEU，并证明pass@k应为神经去编译主评估指标。
原文摘要 · Abstract (English)
Neural decompilation is increasingly studied as a code-generation problem, yet its evaluation methodology remains underdeveloped for modern languages. We present a systematic empirical study of fine-tuning effectiveness and metric validity for Dart Ahead-of-Time (AOT) neural decompilation. We evaluate six fine-tuned model variants across three base architectures (4B-8B parameters) using three metrics: CodeBLEU, compile@k, and pass@k on a new 154-task HumanEval-Dart benchmark. Our study yields three principal findings grounded in paired task-level statistical tests. First, no fine-tuning configuration produces a statistically significant pass@k improvement. The sole positive case yields +0.71 pp (McNemar p=0.21), while fine-tuning the strongest base (Qwen3-8B) causes a highly significant regression of -5.65 pp (p<0.001). This capacity-dependent trend is consistent across architectures but needs broader scale sweeps. Second, cross-lingual interference from Swift training is highly significant at 4B (-2.66 pp, p<0.001) but statistically indistinguishable from zero at 8B, consistent with the scaling hypothesis. Third, we demonstrate metric divergence: CodeBLEU and compile@k can improve significantly while pass@k moves in the opposite direction. This has implications for any LLM code generation task where fine-tuning targets superficial similarity. Error analysis reveals assembly sequence length is the strongest predictor of task difficulty (p=0.001), with a capability cliff at 200 instructions. We contribute the HumanEval-Dart benchmark, a Dart-adapted CodeBLEU, and empirical evidence that pass@k must be the primary evaluation metric for neural decompilation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。