构建混合错误数据集，研究人与大模型代码错误的交互影响。
Tricky$^2$: Towards a Benchmark for Evaluating Human and LLM Error Interactions
- 用分类提示框架在真实代码中注入大模型生成的错误
- 覆盖人类、大模型及混合错误类型，支持多任务评估
- 适合研究人机协作编程或错误检测的开发者
大型语言模型（LLMs）正越来越多地融入软件开发流程，但它们常引入与人类错误不同的细微逻辑或数据误用错误。为研究两类错误的交互，我们构建了Tricky$^2$，一个混合数据集，通过在现有TrickyBugs语料库的人类编写缺陷基础上，注入GPT-5和OpenAI-oss-20b在C++、Python和Java程序中生成的错误。方法采用基于分类体系的提示框架，在保留原始人类缺陷和程序结构的同时生成机器来源的错误。最终数据集包含人类仅有的、大模型仅有的以及人类+大模型混合的三类样本，支持对混合起源错误行为、多错误修复鲁棒性及人机协作代码可靠性进行分析。本文介绍了数据集构建流程，并通过小规模基线评估展示了其在分类、定位和修复任务中的应用。
原文摘要 · Abstract (English)
Large language models (LLMs) are increasingly integrated into software development workflows, yet they often introduce subtle logic or data-misuse errors that differ from human bugs. To study how these two error types interact, we construct Tricky$^2$, a hybrid dataset that augments the existing TrickyBugs corpus of human-written defects with errors injected by both GPT-5 and OpenAI-oss-20b across C++, Python, and Java programs. Our approach uses a taxonomy-guided prompting framework to generate machine-originated bugs while preserving original human defects and program structure. The resulting corpus spans human-only, LLM-only, and human+LLM splits, enabling analysis of mixed-origin error behavior, multi-bug repair robustness, and reliability in hybrid human-machine code. This paper outlines the dataset construction pipeline and illustrates its use through small-scale baseline evaluations of classification, localization, and repair tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。