构建首个完整C转安全Rust的评测基准,评估代码迁移效果。
CRUST-Bench: A Comprehensive Benchmark for C-to-safe-Rust Transpilation
- 用100个真实C项目+手动编写Rust接口与测试用例构建评测集
- 顶尖模型单次尝试仅能正确转换15个任务,仍存严重缺陷
- 适合研究代码翻译、LLM编程能力及系统化迁移工具的开发者
C转Rust的代码移植对现代化遗留C代码至关重要,可提升安全性并增强与现代Rust生态的互操作性。然而,目前尚无数据集用于评估系统是否能将C代码成功转换为通过测试用例验证的安全Rust代码。我们提出CRUST-Bench,包含100个C代码仓库,每个仓库均配有手工编写的安全Rust接口和可用于验证转换正确性的测试用例。通过考虑整个仓库而非孤立函数,该数据集捕捉了跨文件依赖等复杂项目的转换挑战。提供的Rust接口明确了符合习惯且内存安全的模式,测试用例则确保功能正确性。我们在该任务上评估了当前最先进的大语言模型(LLMs),发现生成安全且符合习惯的Rust代码仍是重大挑战。表现最佳的模型OpenAI o1在单次设置下仅能解决15个任务。对CRUST-Bench的改进将推动更强大代码迁移系统的发展,助力从C向保证内存安全的语言如Rust的代码库迁移。数据集与代码可在https://github.com/anirudhkhatry/CRUST-bench获取。
原文摘要 · Abstract (English)
C-to-Rust transpilation is essential for modernizing legacy C code while enhancing safety and interoperability with modern Rust ecosystems. However, no dataset currently exists for evaluating whether a system can transpile C into safe Rust that passes a set of test cases. We introduce CRUST-Bench, a dataset of 100 C repositories, each paired with manually-written interfaces in safe Rust as well as test cases that can be used to validate correctness of the transpilation. By considering entire repositories rather than isolated functions, CRUST-Bench captures the challenges of translating complex projects with dependencies across multiple files. The provided Rust interfaces provide explicit specifications that ensure adherence to idiomatic, memory-safe Rust patterns, while the accompanying test cases enforce functional correctness. We evaluate state-of-the-art large language models (LLMs) on this task and find that safe and idiomatic Rust generation is still a challenging problem for various state-of-the-art methods and techniques. We also provide insights into the errors LLMs usually make in transpiling code from C to safe Rust. The best performing model, OpenAI o1, is able to solve only 15 tasks in a single-shot setting. Improvements on CRUST-Bench would lead to improved transpilation systems that can reason about complex scenarios and help in migrating legacy codebases from C into languages like Rust that ensure memory safety. You can find the dataset and code at https://github.com/anirudhkhatry/CRUST-bench.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。