首个面向师生对话的互动性数据集,助力研究教育对话中的参与度机制。
IntrEx: A Dataset for Modeling Engagement in Educational Conversations
- 基于对比评分法构建序列级标注,捕捉对话中兴趣演变过程。
- 7B/8B参数LLM微调后表现优于GPT-4o,证明专用数据价值。
- 揭示具体语言特征如可理解性、回应率对参与度的影响。
参与度与动机对第二语言习得至关重要,但维持学习者在教育对话中的兴趣仍具挑战。现有研究多关注文本趣味性,却缺乏对对话中驱动参与的语言特征的深入理解。为此,我们提出IntrEx,首个针对师生互动中有趣性及预期有趣的大型标注数据集。该数据集基于教师-学生聊天室语料库(TSCC),通过引入序列级标注,支持对长对话中兴趣动态的分析。采用超过100名二语学习者参与的严谨标注流程,借鉴强化学习人类反馈(RLHF)的对比评分策略以提升标注一致性。我们探究大语言模型(LLMs)预测人类有趣性判断的能力,发现经有趣性评分微调的7B/8B参数模型表现优于更大规模的专有模型如GPT-4o,表明专用数据集在教育场景建模参与度中的潜力。最后,分析了具体语言与认知因素(如具体性、可读性、回应率)如何影响教育对话中的参与度。
原文摘要 · Abstract (English)
Engagement and motivation are crucial for second-language acquisition, yet maintaining learner interest in educational conversations remains a challenge. While prior research has explored what makes educational texts interesting, still little is known about the linguistic features that drive engagement in conversations. To address this gap, we introduce IntrEx, the first large dataset annotated for interestingness and expected interestingness in teacher-student interactions. Built upon the Teacher-Student Chatroom Corpus (TSCC), IntrEx extends prior work by incorporating sequence-level annotations, allowing for the study of engagement beyond isolated turns to capture how interest evolves over extended dialogues. We employ a rigorous annotation process with over 100 second-language learners, using a comparison-based rating approach inspired by reinforcement learning from human feedback (RLHF) to improve agreement. We investigate whether large language models (LLMs) can predict human interestingness judgments. We find that LLMs (7B/8B parameters) fine-tuned on interestingness ratings outperform larger proprietary models like GPT-4o, demonstrating the potential for specialised datasets to model engagement in educational settings. Finally, we analyze how linguistic and cognitive factors, such as concreteness, comprehensibility (readability), and uptake, influence engagement in educational dialogues.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。