将量子编程题库迁移到Qiskit，构建评估大模型量子计算能力的基准测试
Qiskit QuantumKatas: Adapting Microsoft's Quantum Computing exercises for LLM evaluation
- 将微软量子课程移植到Qiskit，设计350个任务覆盖从基础门到高级算法
- 16个大模型测试显示最佳配置通过率32.3%~83.1%，开源模型平均落后26.1个百分点
- 链式思考提示对部分模型提升明显，但整体表现不如少样本提示
我们将微软的QuantumKatas——一个成熟的量子计算教学体系——从Q#迁移至最广泛使用的量子计算框架Qiskit，并配套构建了系统化的LLM评估框架。新基准包含26个类别、共350个任务，涵盖基础门操作到高级算法（如Grover、Simon、Deutsch-Jozsa）、纠错、密钥分发和量子游戏。每个任务均含自然语言提示、标准解法及通过经典电路模拟的确定性验证。基于QuantumKatas已验证的教学设计，我们继承了合理的难度递进与全面的概念覆盖，同时贡献框架适配、评估基础设施与实证分析。我们对16个LLM在7种提示配置下进行评估，总计39,200次模型运行。关键发现：(1) 基准能有效区分模型能力，最优配置通过率32.3%~83.1%，前沿模型与开源模型平均差距26.1个百分点；(2) 模型在实现已知算法上表现良好（Simon算法82.1%，基础门81.6%），但在问题编码上困难重重（用Grover解SAT仅34.4%，区分酉变换40.0%）；(3) 链式思考提示对三款模型有显著提升（其中两款为厂商文档标注的推理优化模型），但对其他模型有负面影响，整体平均通过率56.3%，低于少样本提示5（57.8%）。我们公开基准、评估框架与基线结果，以支持大模型在量子计算领域的能力研究。
原文摘要 · Abstract (English)
We adapt Microsoft's QuantumKatas -- a well-established quantum computing curriculum -- from Q# to Qiskit, the most widely-adopted quantum computing framework, and package it with an evaluation framework for systematic LLM assessment. The resulting benchmark comprises 350 tasks across 26 categories, spanning fundamental gates through advanced algorithms (Grover's, Simon's, Deutsch-Jozsa), error correction, key distribution, and quantum games. Each task includes a natural language prompt, canonical solution, and deterministic test verification via classical circuit simulation. By building on the QuantumKatas' proven pedagogical design rather than creating tasks from scratch, we inherit a principled difficulty progression and comprehensive concept coverage while contributing the framework adaptation, evaluation infrastructure, and empirical analysis. We evaluate 16 LLMs across 7 prompting configurations -- a total of 39,200 model runs -- to demonstrate the benchmark's utility. Three key findings emerge: (1) the benchmark effectively differentiates model capabilities, with best-configuration pass rates ranging from 32.3% to 83.1% and a 26.1 pp average gap between frontier and open-source models; (2) models perform well at implementing known algorithms (SimonsAlgorithm 82.1%, BasicGates 81.6%) but struggle with problem encoding (SolveSATWithGrover 34.4%, DistinguishUnitaries 40.0%); and (3) chain-of-thought prompting shows a modestly bimodal effect -- it is the best strategy for three models (two of them explicitly reasoning-tuned per vendor documentation) but degrades performance for the rest, leaving it mid-pack in aggregate (56.3% mean) behind few-shot-5 (57.8%). We release the benchmark, evaluation framework, and baseline results to support research on LLM capabilities in quantum computing.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。