首个面向多方计算代码修复的专用评测基准,专为安全与数值正确性设计。
MPC-Patch-Bench: Security-Aware LLM Code Patch for Multi-Party Computation

- 构建基于双框架的数据清洗与验证体系,确保修复任务真实可信。
- 仅22.9%最强LLM能正确修复任务,17.1%通过安全验证,40%被拒因加密缺陷。
- 适合关注隐私计算、安全编程或大模型代码修复的研究者使用。
当前缺乏针对多方计算(MPC)软件的仓库级大语言模型(LLM)代码修复评测基准,直接套用通用基准如SWE-bench在三个结构层面失效:(i) MPC仓库以通用Python基础设施为主,非密码逻辑;(ii) 高价值修复缺乏标准化测试;(iii) 传统通过/失败评估不足以衡量需满足密码安全性的代码。随着MPC在隐私保护机器学习、生物医学协作和安全分析中的广泛应用,现有针对特定算子或单框架的任务无法支撑真实仓库级修复评估。我们提出MPC-Patch-Bench,涵盖两个核心组件:(1) 数据清洗框架结合领域特定清洗代理,通过三层密码学过滤并辅以人机协同补全问题描述与失败/通过测试,生成205个完整验证实例;(2) MPC验证器通过动态差分测试对比明文归约器及专属静态分析规则,检测不安全泄露、不安全算术和非法公/私类型转换。最强评估的LLM仅能功能修复22.9%的任务,经验证后降至17.1%,高达40%的功能通过补丁因密码或数值保真度问题被拒绝。
原文摘要 · Abstract (English)
Repository-level benchmarks for evaluating Large Language Model (LLM) code repair on Secure Multi-Party Computation (MPC) software do not yet exist, and directly transplanting general-purpose benchmarks such as SWE-bench fails on three structural fronts: (i) MPC repositories are dominated by generic Python infrastructure rather than cryptographic logic; (ii) high-value MPC fixes lack the standardized tests rigid extraction pipelines require; and (iii) standard fail-to-pass evaluation is insufficient for code that must also be cryptographically safe. MPC is increasingly deployed for privacy-preserving machine learning, biomedical collaboration, and secure analytics. Existing MPC-specific code-synthesis efforts cover only operator-level or single-framework tasks; evaluating LLM agents on real repository-level MPC repair instead demands MPC-aware data curation and a verifier matched to the security and numerical-fidelity guarantees MPC programs must obey neither of which existing benchmarks provide. We introduce MPC-Patch-Bench, a repository-level benchmark organised around two frameworks. (1)The Data Curation Framework combines a domain-specific curation agent that filters raw pull requests through three cryptographic layers with a human-AI completion engine that synthesizes missing problem statements and Fail-to-Pass/Pass-to-Pass tests, yielding 205 fully verified instances. (2)The MPC Verifier provides dedicated security and numerical-fidelity checks via dynamic differential testing against plaintext oracles and MPC-specific static analysis rules that flag unsafe reveals, insecure arithmetic, and illegal public/private casts. The strongest evaluated LLM functionally resolves only 22.9% of MPC-Patch-Bench tasks; the MPC Verifier further reduces verified resolution to 17.1%, with up to 40% of functionally-passing patches rejected for cryptographic or numerical-fidelity violations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。