用逻辑约束分析学生写给大模型的提示词,发现失败者中途频繁改思路。
From Prompts to Propositions: A Logic-Based Lens on Student-LLM Interactions
- 将提示词转化为逻辑约束,量化表达其意图
- 1872条提示中,失败者修改提示更频繁,策略常中途切换
- 可识别学习困难学生,适合教育工具实时干预
大型语言模型(LLM)在计算教育中的应用日益广泛,但如何理解学生使用提示词解决编程任务的行为仍具挑战。现有研究多依赖定性或低效的定量方法,难以捕捉提示词语义的演变过程。本文提出Prompt2Constraints方法,将学生提示词转化为命题逻辑约束,以简洁量化的方式表征其意图。基于对203名学生完成入门编程任务的1,872条提示词数据的分析发现:成功与失败尝试的整体约束数量相似,但失败者往往在过程中显著修改提示,中途转换解题策略。研究还识别出可实施干预的关键节点,为检测学习困难学生提供新路径。该方法具有可扩展性,未来可应用于复杂任务分析,并集成至编程工具实现实时支持。
原文摘要 · Abstract (English)
Background and Context. The increasing integration of large language models (LLMs) in computing education presents an emerging challenge in understanding how students use LLMs and craft prompts to solve computational tasks. Prior research has used both qualitative and quantitative methods to analyze prompting behavior, but these approaches lack scalability or fail to effectively capture the semantic evolution of prompts. Objective. In this paper, we investigate whether students prompts can be systematically analyzed using propositional logic constraints. We examine whether this approach can identify patterns in prompt evolution, detect struggling students, and provide insights into effective and ineffective strategies. Method. We introduce Prompt2Constraints, a novel method that translates students prompts into logical constraints. The constraints are able to represent the intent of the prompts in succinct and quantifiable ways. We used this approach to analyze a dataset of 1,872 prompts from 203 students solving introductory programming tasks. Findings. We find that while successful and unsuccessful attempts tend to use a similar number of constraints overall, when students fail, they often modify their prompts more significantly, shifting problem-solving strategies midway. We also identify points where specific interventions could be most helpful to students for refining their prompts. Implications. This work offers a new and scalable way to detect students who struggle in solving natural language programming tasks. This work could be extended to investigate more complex tasks and integrated into programming tools to provide real-time support.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。