测试大模型生成量子代码时对SDK版本的准确性，发现版本兼容性问题普遍且难修复。
Benchmarking API Drift in LLM-Generated Quantum Code Across Successive SDK Versions
- 构建量子SDK版本漂移基准，评估代码在不同版本中能否正确执行
- 16个模型在3个版本上平均通过率仅0.02至0.85，强模型也常出现弃用错误
- 文档引导修复对升级到v2.0有效，但对v1.3效果差，版本迁移仍不完全可靠
大型语言模型可生成看似合理的量子代码，但其能否准确适配用户指定的软件开发工具包（SDK）版本尚不明确。本文将此问题定义为API漂移，提出quantum-api-drift基准，用于衡量版本保真度——即在指定SDK版本上的执行成功率、跨版本兼容性、失败模式及基于文档的修复能力。以经历重大接口变更的Qiskit为例，在v0.43、v1.3、v2.0三个版本上评估17个模型，每个任务生成3份样本，共450个生成样本，每模型执行1,350次。16个模型在匹配的REST API设置下（输出限制1024词元），其Pass@1得分在0.02至0.85之间。Claude Opus 4.7在v0.43和v2.0表现最佳，Grok 4.20在v1.3上为0.513。弱模型多因导入错误失败，强模型则更常见弃用级错误。基于文档的修复整体成功率为0.19至0.59，迁移至v2.0比至v1.3更有效。相关数据已公开于https://github.com/arasyi/quantum-api-drift。结果表明，版本对齐是量子代码生成的关键评估维度，即使有迁移指导，API漂移仍难以完全恢复。
原文摘要 · Abstract (English)
Large language models can generate plausible quantum code, but it is unclear whether they can reliably target the specific software development kit (SDK) version requested by the user. We study this problem as API drift and introduce quantum-api-drift, a benchmark for measuring version fidelity, defined here as execution success on the requested SDK version, cross-version compatibility, failure modes, and documentation-guided repair in LLM-generated quantum SDK code. We instantiate the benchmark with Qiskit, a representative quantum SDK that underwent substantial interface changes across v0.43, v1.3, and v2.0. We evaluate 17 models on 50 tasks with 3 samples per prompt, yielding 450 generated samples and 1,350 executions per model. Sixteen models are tested in a matched REST API setting with a 1024-token output cap, while GPT-5.4 (Codex CLI) is reported separately as a reference configuration. Across the 16 matched REST models, diagonal Pass@1 ranges from 0.02 to 0.85. Claude Opus 4.7 is strongest on v0.43 and v2.0, while Grok 4.20 is strongest on v1.3 at 0.513. Error profiles differ systematically by model strength: weaker models fail mainly with broken imports, while stronger models more often reach deprecation-level failures. Documentation-guided repair succeeds for 0.19 to 0.59 of repair attempts overall and is consistently much more effective for migration to v2.0 than to v1.3. The benchmark artifacts are publicly available at https://github.com/arasyi/quantum-api-drift. These results show that version alignment is a distinct evaluation axis for quantum code generation and that API drift remains only partly recoverable even with migration guidance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。