arXiv:2503.19279cs.CLcs.LG2025-03被引 7

用预训练模型自动分析写作中的论点结构,提升英语作文评价效率

Machine-assisted writing evaluation: Exploring pre-trained language models in analyzing argumentative moves

  • 用BERT模型自动识别六类论点成分,准确率达F1 0.743
  • 发现学生随时间增加使用证据和反论点,低分作文多单边论据
  • 适合教育科技开发者和语言教学研究者参考

本研究探讨预训练语言模型(PLMs)在分析中国235名英语学习者历时性议论文中的论点结构的有效性。收集了1643篇作文,划分为六个论点类型:主张、证据、反主张、反证据、反驳和非论点。采用人类专家与PLM双重标注,以BERT为实现模型之一。结果表明,PLM在识别论点结构上具有强可靠性,整体F1得分为0.743,优于现有模型。此外,基于PLM标注的论点分布能有效捕捉学生写作发展规律:高阶写作者更频繁使用反主张、反证据与反驳,低质量文本则集中于单一立场的主张与证据。该研究证明人工智能可显著提升写作评估效率与精准度,推动教育技术向数据驱动与个性化方向发展。

原文摘要 · Abstract (English)

The study investigates the efficacy of pre-trained language models (PLMs) in analyzing argumentative moves in a longitudinal learner corpus. Prior studies on argumentative moves often rely on qualitative analysis and manual coding, limiting their efficiency and generalizability. The study aims to: 1) to assess the reliability of PLMs in analyzing argumentative moves; 2) to utilize PLM-generated annotations to illustrate developmental patterns and predict writing quality. A longitudinal corpus of 1643 argumentative texts from 235 English learners in China is collected and annotated into six move types: claim, data, counter-claim, counter-data, rebuttal, and non-argument. The corpus is divided into training, validation, and application sets annotated by human experts and PLMs. We use BERT as one of the implementations of PLMs. The results indicate a robust reliability of PLMs in analyzing argumentative moves, with an overall F1 score of 0.743, surpassing existing models in the field. Additionally, PLM-labeled argumentative moves effectively capture developmental patterns and predict writing quality. Over time, students exhibit an increase in the use of data and counter-claims and a decrease in non-argument moves. While low-quality texts are characterized by a predominant use of claims and data supporting only oneside position, mid- and high-quality texts demonstrate an integrative perspective with a higher ratio of counter-claims, counter-data, and rebuttals. This study underscores the transformative potential of integrating artificial intelligence into language education, enhancing the efficiency and accuracy of evaluating students' writing. The successful application of PLMs can catalyze the development of educational technology, promoting a more data-driven and personalized learning environment that supports diverse educational needs.

写作评估预训练模型教育技术

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。