提出细粒度评估框架,实现跨语言代码库精准翻译与自动测试验证
Skeleton-Guided-Translation: A Benchmarking Framework for Code Repository Translation with Fine-Grained Quality Evaluation
- 分两步翻译:先提取代码结构骨架,再基于骨架生成完整代码
- 构建了高质量开源代码库基准集,支持自动化测试与增量翻译
- 按单个测试用例评估质量,突破传统二元评价局限
大型语言模型的发展推动企业应用现代化与遗留系统向安全、通用语言迁移。然而现有代码翻译基准多聚焦单个函数,忽略整个代码库翻译中的模块间一致性与依赖管理难题。尽管部分近期工作尝试解决,仍存在可维护性差、评估粒度粗等问题。本文提出Skeleton-Guided-Translation框架,用于Java到C#的代码库级翻译,采用两阶段流程:先翻译代码结构‘骨架’,再以骨架为引导完成全库翻译。基于此,构建TRANSREPO-BENCH基准,包含高质量开源Java代码库及其对应C#骨架,附带可复用的单元测试和构建配置。测试用例经修复可直接应用于多次或增量翻译,提升评估自动化与可扩展性。同时开发细粒度评估指标,可在单个测试案例层面衡量翻译质量,克服传统二元指标无法区分因编译失败导致全部测试失败的问题。实验表明该框架揭示关键挑战,推动更精准的代码库级翻译发展。
原文摘要 · Abstract (English)
The advancement of large language models has intensified the need to modernize enterprise applications and migrate legacy systems to secure, versatile languages. However, existing code translation benchmarks primarily focus on individual functions, overlooking the complexities involved in translating entire repositories, such as maintaining inter-module coherence and managing dependencies. While some recent repository-level translation benchmarks attempt to address these challenges, they still face limitations, including poor maintainability and overly coarse evaluation granularity, which make them less developer-friendly. We introduce Skeleton-Guided-Translation, a framework for repository-level Java to C# code translation with fine-grained quality evaluation. It uses a two-step process: first translating the repository's structural "skeletons", then translating the full repository guided by these skeletons. Building on this, we present TRANSREPO-BENCH, a benchmark of high quality open-source Java repositories and their corresponding C# skeletons, including matching unit tests and build configurations. Our unit tests are fixed and can be applied across multiple or incremental translations without manual adjustments, enhancing automation and scalability in evaluations. Additionally, we develop fine-grained evaluation metrics that assess translation quality at the individual test case level, addressing traditional binary metrics' inability to distinguish when build failures cause all tests to fail. Evaluations using TRANSREPO-BENCH highlight key challenges and advance more accurate repository level code translation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。