用大模型+图约束自动标注学习资源的胜任力,既准又可追溯。
From Learning Resources to Competencies: LLM-Based Tagging with Evidence and Graph Constraints

- 大模型结合图结构上下文,从教学内容中精准提取胜任力标签。
- 片段级微平均F1达0.57,资源级宏平均F1达0.51,排序指标MRR为0.82。
- 生成可追踪的证据片段,适合需透明性与教育分析的系统应用。
将学习资源与结构化胜任力框架对齐,是实现学习管理系统中基于胜任力的检索与课程分析的关键。然而,手动标注耗时,全自动方法常缺乏透明度。本文提出一个端到端对齐流程,使用大语言模型(LLM)作为受约束、可生成证据的标签器。教学内容和评估材料首先被分割为有意义的教学片段。针对每个片段,从富含图结构上下文的胜任力档案中检索一组候选胜任力。随后,LLM从该集合中选出最相关胜任力,并从文本中提取支持性证据片段。这些预测通过胜任力图结构进行优化,并在资源层级聚合。我们在法国贡比涅技术大学计算机科学系的胜任力参考体系上构建数据集,涵盖22项胜任力及多门课程材料。所提出的LLM+BM25+Graph(LBG)管道在片段级取得0.57的微平均F1和0.50的宏平均F1,资源级宏平均F1为0.51,MRR达0.82,优于零样本、少样本LLM变体、检索/相似性基线及监督分类器,同时生成更可机械追踪的证据片段,便于人工审计与教育分析。
原文摘要 · Abstract (English)
Linking learning resources to a structured competency framework is key to enabling competency-based search and curriculum analytics in Learning Management Systems (LMS). However, manual tagging is labor-intensive, and fully automatic methods often lack transparency. In this paper, we present an end-to-end alignment pipeline that uses a large language model (LLM) as a constrained, evidence-producing tagger. LMS resources -both instructional content and assessments -are first segmented into meaningful pedagogical fragments. For each fragment, a small set of candidate competencies is retrieved from structured competency profiles enriched with graph-based context. The LLM then selects the most relevant competencies from this set and provides supporting evidence spans from the fragment text. These predictions are refined using the structure of the competency graph and aggregated at the resource level. We evaluate our approach on a dataset built from the Computer Science department's competency referential at the Université de Technologie de Compiègne (UTC), covering 22 competencies across multiple course materials. Our LLM+BM25+Graph (LBG) pipeline achieves strong results, with a micro-F1 of 0.57 and macro-F1 of 0.50 at the fragment level, 0.51 macro-F1 at the resource level, and an MRR of 0.82outperforming zero-shot and few-shot LLM variants, retrieval/similarity baselines, and supervised classifiers -while also producing more mechanically traceable evidence spans to support human auditing and educational analysis.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。