用领域适配检索提升教学对话标注准确率,不改大模型也能显著提效。
Domain-Adapted Retrieval for In-Context Annotation of Pedagogical Dialogue Acts
- 通过微调轻量嵌入模型,实现教学语料的逐句索引与少样本示例召回。
- 在两个真实教学数据集上,标注一致性κ值提升至0.526-0.743,远超无检索基线。
- 特别改善罕见和依赖上下文的标签,且无需微调生成模型,适合快速部署。
自动标注教学对话是一项高风险任务,大语言模型常因缺乏领域知识而失效。本文提出一种领域适配的RAG流水线,用于教学行为标注。不微调生成模型,而是通过在教学语料上微调轻量级嵌入模型,并以逐句级别索引对话,召回带标签的少样本示范。在两个真实教学对话数据集(TalkMoves和Eedi)及三种LLM基座(GPT-5.2、Claude Sonnet 4.6、Qwen3-32b)上评估,最佳配置在TalkMoves上达到Cohen's κ 0.526–0.580,Eedi上达0.659–0.743,显著优于无检索基线(κ=0.275–0.413 和 0.160–0.410)。消融实验表明,逐句索引是主要增益来源,而非嵌入质量本身;在TalkMoves上,最优先验匹配率从39.7%提升至62.0%,Eedi上从52.9%升至73.1%。检索还能纠正零样本提示中的系统性标签偏差,对罕见和上下文依赖标签提升最大。结果表明,仅适配检索组件即可实现专家级教学对话标注,同时保持生成模型冻结。
原文摘要 · Abstract (English)
Automated annotation of pedagogical dialogue is a high-stakes task where LLMs often fail without sufficient domain grounding. We present a domain-adapted RAG pipeline for tutoring move annotation. Rather than fine-tuning the generative model, we adapt retrieval by fine-tuning a lightweight embedding model on tutoring corpora and indexing dialogues at the utterance level to retrieve labeled few-shot demonstrations. Evaluated across two real tutoring dialogue datasets (TalkMoves and Eedi) and three LLM backbones (GPT-5.2, Claude Sonnet 4.6, Qwen3-32b), our best configuration achieves Cohen's $κ$ of 0.526-0.580 on TalkMoves and 0.659-0.743 on Eedi, substantially outperforming no-retrieval baselines ($κ= 0.275$-$0.413$ and $0.160$-$0.410$). An ablation study reveals that utterance-level indexing, rather than embedding quality alone, is the primary driver of these gains, with top-1 label match rates improving from 39.7\% to 62.0\% on TalkMoves and 52.9\% to 73.1\% on Eedi under domain-adapted retrieval. Retrieval also corrects systematic label biases present in zero-shot prompting and yields the largest improvements for rare and context-dependent labels. These findings suggest that adapting the retrieval component alone is a practical and effective path toward expert-level pedagogical dialogue annotation while keeping the generative model frozen.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。