首个跨框架量子代码生成基准，评估大模型在三类工具中的表现。
QuanBench+: A Unified Multi-Framework Benchmark for LLM-Based Quantum Code Generation
- 构建统一评测框架，覆盖Qiskit、PennyLane、Cirq三大量子开发工具。
- 单次生成最高通过率59.5%（Qiskit），反馈修复后提升至83.3%。
- 揭示模型仍严重依赖特定框架知识，通用能力仍有短板。
大型语言模型在代码生成中应用日益广泛，但量子代码生成仍主要局限于单一框架，难以区分量子推理能力与框架熟悉度。我们提出QuanBench+，一个覆盖Qiskit、PennyLane和Cirq的统一基准，包含42个对齐任务，涵盖量子算法、门分解和态制备。采用可执行功能测试评估模型，报告Pass@1和Pass@5指标，并使用基于KL散度的接受准则处理概率输出。此外，研究了基于反馈修复后的Pass@1表现，即模型在运行时错误或答案错误后修正代码的能力。跨框架下，最强的一次生成得分分别为：Qiskit 59.5%、Cirq 54.8%、PennyLane 42.9%；经反馈修复后，最佳得分分别提升至83.3%、76.2%、66.7%。结果表明进展明显，但可靠的多框架量子代码生成仍未解决，仍高度依赖框架特定知识。
原文摘要 · Abstract (English)
Large Language Models (LLMs) are increasingly used for code generation, yet quantum code generation is still evaluated mostly within single frameworks, making it difficult to separate quantum reasoning from framework familiarity. We introduce QuanBench+, a unified benchmark spanning Qiskit, PennyLane, and Cirq, with 42 aligned tasks covering quantum algorithms, gate decomposition, and state preparation. We evaluate models with executable functional tests, report Pass@1 and Pass@5, and use KL-divergence-based acceptance for probabilistic outputs. We additionally study Pass@1 after feedback-based repair, where a model may revise code after a runtime error or wrong answer. Across frameworks, the strongest one-shot scores reach 59.5% in Qiskit, 54.8% in Cirq, and 42.9% in PennyLane; with feedback-based repair, the best scores rise to 83.3%, 76.2%, and 66.7%, respectively. These results show clear progress, but also that reliable multi-framework quantum code generation remains unsolved and still depends strongly on framework-specific knowledge.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。