首个面向OpenQASM-3硬件特性的大模型训练评估数据集
QASM-Eval: A Dataset to Train and Evaluate LLMs on OpenQASM-3 Beyond Quantum Circuits
- 构建涵盖经典逻辑、时序调度等硬件特性的4000个任务训练集
- 100个专家验证任务中，现役大模型生成准确率不足20%
- 专为量子纠错、脉冲控制等真实场景设计，适配量子编程助手研发
量子计算仍处于噪声中等规模量子（NISQ）时代，性能受限于噪声。解决此类限制通常需要超越门序列电路描述的硬件级能力，包括中间测量与经典反馈用于量子误差纠正（QEC）、动态解耦（DD）的精确时序控制，以及用于校准的脉冲级波形访问。OpenQASM-3被引入以暴露这些能力，提供硬件级编程接口。然而，尽管大语言模型在代码生成方面进展迅速，目前仍缺乏专门针对包含高级硬件特性的OpenQASM-3程序进行训练与评估的数据集。为此，我们提出QASM-Eval，首个专为大模型在OpenQASM-3上训练与评估设计的综合性数据集。该数据集不聚焦于量子算法设计或推理，而是明确针对语言的硬件面向特性。QASM-Eval包含经专家验证的100个测试任务和4000个训练任务，系统覆盖经典逻辑、时序调度、脉冲控制及复杂真实工作流。为自动验证生成程序，我们使用扩展验证器检查语法、量子态及程序时间线。评估显示，当前最先进大模型在OpenQASM-3编码任务中表现不佳，但在QASM-Eval上针对性微调后取得显著提升。QASM-Eval为加速NISQ时代硬件级量子编程大模型助手的可靠发展提供了关键基准与训练基础。数据与代码：https://github.com/fuzhenxiao/QASM-Eval
原文摘要 · Abstract (English)
Quantum computing remains in the Noisy Intermediate-Scale Quantum (NISQ) era, where the performance is highly constrained to noise. Addressing the limitation often requires hardware-facing capabilities beyond gate-sequence circuit specification, including mid-circuit measurement and classical feedback for quantum error correction (QEC), precise timing control for dynamical decoupling (DD), and pulse-level waveform access for calibration. OpenQASM-3 was introduced to expose exactly these capabilities, providing a hardware-level programming interface. However, despite the rapid progress of large language models in code generation, there is still no dataset specifically designed to train and evaluate LLMs on OpenQASM-3 programs that involve its advanced hardware-oriented features. To address this gap, we introduce QASM-Eval, the first comprehensive dataset designed to train and evaluate LLMs on OpenQASM-3. Rather than focusing on quantum algorithm design or reasoning, QASM-Eval explicitly targets the language's hardware-facing features. QASM-Eval comprises an expert-verified test set of 100 tasks and a training set of 4,000 tasks, systematically covering classical logic, timing scheduling, pulse control, and complex real-world workflows. To automatically validate generated programs, we check syntax, quantum states and program timeline using an extended verifier. Our evaluation reveals that while state-of-the-art LLMs struggle heavily in OpenQASM-3 coding tasks, targeted fine-tuning on QASM-Eval yields significant gains. QASM-Eval provides a crucial benchmark and training foundation to accelerate the development of reliable LLM assistants for hardware-facing quantum programming in NISQ era. Data and code: https://github.com/fuzhenxiao/QASM-Eval
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。