构建4112个编程任务数据集，用大模型自动判断任务难度。
TaskComplexity: A Dataset for Task Complexity Classification with In-Context Learning, FLAN-T5 and GPT-4o Benchmarks
- 从多个网站爬取编程题，提取题目描述与复杂度标签。
- GPT-4o-mini用上下文学习比微调FLAN-T5准确率更高。
- 适合做编程任务分派或智能导师系统的研究者参考。
本文针对编程任务分类与专家分配耗时耗力的问题，构建了一个包含4,112个编程任务的新数据集，通过网络爬虫系统性地从多个网站提取任务。利用特定的HTML标签，提取了每道题的标题、问题描述、输入输出格式、示例、题目类别和复杂度评分等关键信息。数据集效果通过两种方法评估：一是对FLAN-T5 small模型进行微调，二是使用GPT-4o-mini进行上下文学习（ICL）。采用准确率、召回率、精确率和F1分数作为标准指标。结果表明，基于GPT-4o-mini的上下文学习方法在性能上优于微调后的FLAN-T5模型。
原文摘要 · Abstract (English)
This paper addresses the challenge of classifying and assigning programming tasks to experts, a process that typically requires significant effort, time, and cost. To tackle this issue, a novel dataset containing a total of 4,112 programming tasks was created by extracting tasks from various websites. Web scraping techniques were employed to collect this dataset of programming problems systematically. Specific HTML tags were tracked to extract key elements of each issue, including the title, problem description, input-output, examples, problem class, and complexity score. Examples from the dataset are provided in the appendix to illustrate the variety and complexity of tasks included. The dataset's effectiveness has been evaluated and benchmarked using two approaches; the first approach involved fine-tuning the FLAN-T5 small model on the dataset, while the second approach used in-context learning (ICL) with the GPT-4o mini. The performance was assessed using standard metrics: accuracy, recall, precision, and F1-score. The results indicated that in-context learning with GPT-4o-mini outperformed the FLAN-T5 model.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。