测试ChatGPT-3.5在编程教学中的表现，发现越专业越容易出错。
Benefits and Risks of Using ChatGPT4 as a Teaching Assistant for Computer Science Students
- 用标准化问题测试模型在算法、设计模式、量子计算三领域的回答质量。
- 基础算法正确率高，设计模式代码有缺陷但可修复，量子计算常出现严重错误。
- 适合教育者了解AI助教的边界，警惕高阶内容中的误导风险。
ChatGPT-3.5发布后，其解答编程专业问题的能力震惊了软件工程界。许多教育者开始思考是否可将其作为学生编程答疑的支持工具。本文从基础计算机科学知识（基本算法与数据结构）、核心能力（设计模式）和高级知识（量子计算）三个层面评估其可行性。通过多次提出标准化问题，检查答案正确性及潜在问题。结果表明：在基础算法领域，回答几乎全部正确；在设计模式领域，生成代码存在大量代码异味，整体质量较低，但若进一步提问仍可修复；而在量子计算领域，模型经常给出明显错误的答案。
原文摘要 · Abstract (English)
Upon release, ChatGPT3.5 shocked the software engineering community by its ability to generate answers to specialized questions about coding. Immediately, many educators wondered if it was possible to use the chatbot as a support tool that helps students answer their programming questions. This article evaluates this possibility at three levels: fundamental Computer Science knowledge (basic algorithms and data structures), core competency (design patterns), and advanced knowledge (quantum computing). In each case, we ask normalized questions several times to ChatGPT3.5, then look at the correctness of answers, and finally check if this creates issues. The main result is that the performances of ChatGPT3.5 degrades drastically as the specialization of the domain increases: for basic algorithms it returns answers that are almost always correct, for design patterns the generated code contains many code smells and is generally of low quality, but it is still sometimes able to fix it (if asked), and for quantum computing it is often blatantly wrong.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。