对比AI聊天机器人与传统搜索在课程学习中的效果与偏好。
Comparing the Utility, Preference, and Performance of Course Material Search Functionality and Retrieval-Augmented Generation Large Language Model (RAG-LLM) AI Chatbots in Information-Seeking Tasks
- 用课程资料增强大模型生成答案，构建智能助教。
- 14名学生测试显示两种工具各有所长，但性能相近。
- 先用哪个工具影响偏好，后用者更倾向其功能。
为应对学生数量增长带来的支持压力，本研究探索大型语言模型（LLM）作为学习支持工具的潜力。我们开发了一个基于课程资料增强的答案生成型AI聊天机器人（RAG-LLM），并开展实验室用户研究（N=14），参与者完成一个网络软件开发课程的任务。两组学生分别先使用聊天机器人或传统搜索功能，再切换至另一工具。结果表明，两种支持方式均被视作有用，且对特定任务表现良好，但对其他任务效果有限。学生偏好受使用顺序影响：先用聊天机器人的学生更倾向搜索功能，反之亦然。
原文摘要 · Abstract (English)
Providing sufficient support for students requires substantial resources, especially considering the growing enrollment numbers. Students need help in a variety of tasks, ranging from information-seeking to requiring support with course assignments. To explore the utility of recent large language models (LLMs) as a support mechanism, we developed an LLM-powered AI chatbot that augments the answers that are produced with information from the course materials. To study the effect of the LLM-powered AI chatbot, we conducted a lab-based user study (N=14), in which the participants worked on tasks from a web software development course. The participants were divided into two groups, where one of the groups first had access to the chatbot and then to a more traditional search functionality, while another group started with the search functionality and was then given the chatbot. We assessed the participants' performance and perceptions towards the chatbot and the search functionality and explored their preferences towards the support functionalities. Our findings highlight that both support mechanisms are seen as useful and that support mechanisms work well for specific tasks, while less so for other tasks. We also observe that students tended to prefer the second support mechanism more, where students who were first given the chatbot tended to prefer the search functionality and vice versa.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。