用教育理论指导大模型,精细评估本科论文的六个维度。
PEMUTA: Pedagogically-Enriched Multi-Granular Undergraduate Thesis Assessment
- 基于维果斯基和布卢姆理论,分六层评估论文结构、逻辑等
- 在真实论文数据集上与专家评分高度一致,无需微调
- 适合需要精准反馈的高校导师和论文评审场景
本科毕业论文(UGTE)是衡量学生大学阶段综合学术发展的重要环节。尽管大语言模型(LLMs)已推动教育智能化,但通常仅提供单一评分,忽略多维度评估的复杂性,难以反映结构标准、教学目标和多元学术能力。而教育理论长期指导人工评审,涵盖认知发展、学科思维和学业表现,却未被充分应用于自动化系统。为此,我们提出PEMUTA框架,通过激活大模型中的领域知识,实现多粒度的UGTE评估。该框架基于维果斯基理论和布卢姆分类学,采用分层提示策略,从结构、逻辑、原创性、写作、专业性和严谨性(SLOWPR)六个细粒度维度进行评估,并进行整体合成。引入少样本提示和角色扮演提示两种上下文学习技术,增强与专家判断的一致性,无需微调。我们构建了包含真实本科生论文及专家标注SLOWPR标签的数据集,支持多粒度评估。大量实验表明,PEMUTA在与专家评分对齐方面表现优异,展现出在细粒度、教育导向评估中的巨大潜力。
原文摘要 · Abstract (English)
The undergraduate thesis (UGTE) plays an indispensable role in assessing a student's cumulative academic development throughout their college years. Although large language models (LLMs) have advanced education intelligence, they typically focus on holistic assessment with only one single evaluation score, but ignore the intricate nuances across multifaceted criteria, limiting their ability to reflect structural criteria, pedagogical objectives, and diverse academic competencies. Meanwhile, pedagogical theories have long informed manual UGTE evaluation through multi-dimensional assessment of cognitive development, disciplinary thinking, and academic performance, yet remain underutilized in automated settings. Motivated by the research gap, we pioneer PEMUTA, a pedagogically-enriched framework that effectively activates domain-specific knowledge from LLMs for multi-granular UGTE assessment. Guided by Vygotsky's theory and Bloom's Taxonomy, PEMUTA incorporates a hierarchical prompting scheme that evaluates UGTEs across six fine-grained dimensions: Structure, Logic, Originality, Writing, Proficiency, and Rigor (SLOWPR), followed by holistic synthesis. Two in-context learning techniques, \ie, few-shot prompting and role-play prompting, are also incorporated to further enhance alignment with expert judgments without fine-tuning. We curate a dataset of authentic UGTEs with expert-provided SLOWPR-aligned annotations to support multi-granular UGTE assessment. Extensive experiments demonstrate that PEMUTA achieves strong alignment with expert evaluations, and exhibits strong potential for fine-grained, pedagogically-informed UGTE evaluations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。