arXiv:2605.29054cs.SEcs.CL2026-05

代码转换需通过严格等价性验证,避免表面正确却语义错误。

Converted, Not Equivalent: Benchmarking Codebase Conversion via Observational Equivalence

论文配图:Converted, Not Equivalent: Benchmarking Codebase Conversion via Observational Equivalence
图 1 · 摘自论文原文
  • 以固定等价契约重构代码转换评估,分三阶段验证接口、数值和行为一致性
  • 最佳系统仅26.7%~28.9%通过率,虽接口通过率达91.1%但整体失效
  • 自验证机制严重高估成果,适合关注模型可靠性与验证设计的研究者

编码代理在代码库级协作中日益普及,但存在严重缺陷:过度依赖局部验证,误判仅满足表面检查却违背用户实际语义需求的产物。此问题在代码库转换中尤为突出,因以往评估以结果为导向,极不稳定——两个实现可能在前向损失等浅层输出上一致,却在梯度、优化器行为或短期训练动态上严重偏离。我们提出T2J-Bench,一种基于固定等价契约的代码库转换基准。固定验证器通过三个有序阶段对比源码与转换后代码:Spec(接口合规性)、Numeric(前向输出、损失、梯度及特定目标张量)、Behavioral(固定种子下的短期训练动态)。在355次盲转换尝试中,最优系统整体通过率仅为26.7%–28.9%,尽管Spec通过率最高达91.1%;4.7倍令牌预算差异仅带来2.2倍通过率差异;所有系统相对于固定评估器高估成功程度达66.6–97.8分。表明失败主因是自验证与等价契约不匹配,而非资源限制或主干模型强度。

原文摘要 · Abstract (English)

Coding agents increasingly act as codebase-scale collaborators that can assist with codebase conversion, but this progress has exposed a critical weakness: agents often over-trust their own local validation routines and declare success on artifacts that satisfy surface checks while violating the semantic contracts users actually care about. This problem is especially acute in codebase conversion, where prior evaluation is largely outcome-driven and therefore unstable: two implementations can match on a shallow outcome, such as a single forward loss, while diverging in gradients, optimizer behavior, or short-horizon training dynamics. We introduce T2J-Bench, a benchmark for codebase conversion that reformulates conversion as transfer under a fixed equivalence contract. A fixed verifier then compares source and converted codebases through three ordered stages: Spec (interface admissibility), Numeric (forward outputs, losses, gradients, and objective-specific tensors), and Behavioral (short training dynamics under fixed seeds). Across 355 blind conversion attempts, the best system reaches only 26.7--28.9% overall pass rate despite Spec pass rates up to 91.1%; a 4.7x token-budget spread yields only a 2.2x pass-rate spread; and all systems overestimate success by 66.6--97.8 points relative to the fixed evaluator. This suggests that failures stem more from contract-misaligned self-validation than from limited budget or backbone strength.

代码转换验证基准等价性测试

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。