arXiv:2502.03806cs.SEcs.LG2025-02中稿 · the 22nd Internati…被引 2

用代码长度和环复杂度设计教学进度,发现代码模型训练易失效。

Should Code Models Learn Pedagogically? A Preliminary Evaluation of Curriculum Learning for Real-World Software Engineering Tasks

  • 按代码长度与环复杂度分阶训练,测试教学进度法效果
  • 模型仅前25%训练就饱和,出现灾难性遗忘与捷径学习
  • 适合研究代码模型训练机制或教学策略的开发者参考

基于学习的代码模型在代码理解与生成任务中表现优异,但当前训练通常随机打乱数据,可能未达最优。尽管已有研究显示教学进度学习(CL)可通过合成代码难度递增提升性能,但其在真实软件工程任务中使用传统难度指标的有效性仍不明确。本研究采用代码长度和环复杂度作为难度衡量标准,评估预训练代码模型CodeT5在代码克隆检测与代码摘要任务中的表现。在CodeXGLUE基准上的实证研究发现,模型出现灾难性遗忘与捷径学习,且性能在完成首个四分之一训练后即趋于饱和,暗示模型表征能力或任务内在难度存在局限。未来需在更多模型与任务上探索不同教学策略以深化理解。

原文摘要 · Abstract (English)

Learning-based techniques, especially advanced pre-trained models for code have demonstrated capabilities in code understanding and generation, solving diverse software engineering (SE) tasks. Despite the promising results, current training approaches may not fully optimize model performance, as they typically involve learning from randomly shuffled training data. Recent work shows that Curriculum Learning (CL) can improve performance on code-related tasks through incremental learning based on the difficulty of synthetic code. Yet, the effectiveness of CL with conventional difficulty measures in SE tasks remains largely unexplored. In this study, we explore two conventional code metrics: code length and cyclomatic complexity to determine the difficulty levels. We investigate how the pre-trained code model (CodeT5) learns under CL, through the tasks of code clone detection and code summarization. Our empirical study on the CodeXGLUE benchmark showed contrasting results to prior studies, where the model exhibited signs of catastrophic forgetting and shortcut learning. Surprisingly, model performance saturates after only the first quartile of training, potentially indicating a limit in the model's representation capacity and/or the task's inherent difficulty. Future work should further explore various CL strategies with different code models across a wider range of SE tasks for a more holistic understanding.

代码模型教学进度训练优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。