让中学生参与评估模型,让他们的声音被真正理解。
When Youth Enter The Chat: An Epistemic Shift in the Validation of LLM-Based Measures of Student Talk

- 用民族志方法重置课堂对话语境,让学生成为知识生产主体。
- 学生对AI分析结果提出质疑,显示算法分类与真实体验不符。
- 适合关注教育公平、青少年话语权的教育科技研究者。
大语言模型正被广泛用于大规模评估学生课堂讨论(如发言行为、协作、话语权平衡)。当前做法通常仅基于口语转录文本进行分析,脱离真实语境。验证方式多依赖成人专家标注、留出集和F1分数,但这些方法难以确保评估对少数族裔和语言边缘化学生具有意义与公平性。本文主张必须将青年置于分析中心,通过重新嵌入对话语境并让他们参与研究过程,共享认识论权威。在一所八年级数学课堂的案例研究中,采用参与式观察、访谈、焦点小组和成员核对等方法,与四位核心学生合作。研究发现,学生对自己数学对话的理解与其被模型标记的结果存在显著偏差,他们直接挑战了模型分类与编码方案,凸显青年必须深度参与关于自身学习经验的知识建构过程。
原文摘要 · Abstract (English)
LLMs are being used increasingly to measure aspects of student discourse (e.g. talk moves, collaboration, equity of voice) at scale. Typically, LLM-based measures of student talk use transcriptions of classroom conversations that only include verbal contributions, which de-contextualize student language. Common practices for validating these measures include comparing outputs against expert annotations by adults, using held out evaluation sets and F1 scores. We argue that these approaches are insufficient to ensure that such measures are meaningful and equitable for teaching and learning, particularly for racially and linguistically marginalized youth. In order to center the youth whose talk is being analyzed, re-contextualizing these classroom conversations and engaging youth in the research process is necessary. Sharing epistemic authority with youth, ultimately, centers their point of view and adds crucial nuance to the analysis of their talk that adult experts, researchers, and LLMs cannot provide. In a case study of multilingual youth in one 8th-grade math classroom, we address the epistemic exclusion of youth by employing multiple ethnographically-oriented methods to re-contextualize student conversations and center youth as epistemic authorities in conversation with researchers and LLMs. We conducted participant observations, interviews, focus groups, and member checks with four focal students. Findings reveal that there were misalignments between students' interpretations of their own math talk experiences and the LLM-based measures of their talk. Students contested both the LLM classifications and the coding scheme used to measure their talk, highlighting the need for youth to be involved in the epistemic process of producing knowledge about their experiences.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。