用图文互动方式教数学题，让学习更直观高效。
Interactive Sketchpad: A Multimodal Tutoring System for Collaborative, Visual Problem-Solving
- 结合语言与动态图形提供分步指导
- 通过代码生成精准几何图示
- 适合需要视觉化解题的师生使用
人类长期依赖草图和图表辅助推理与解题，如几何中的辅助线或微积分中的函数图像，是理解复杂概念的关键工具。然而，多数辅导系统仍以纯文本形式反馈，仅依赖自然语言。本文基于预训练的大型多模态模型（LMM），提出Interactive Sketchpad——一个融合语言解释与交互式可视化的新一代辅导系统。该系统经过微调，可同时输出文字说明与动态图形，支持学生自然的多模态互动。通过将代码执行嵌入推理流程，确保生成图示的准确性与鲁棒性。在几何、微积分和三角学等数学问题上的用户研究显示，该系统显著提升了任务理解度、解题准确率与学习参与度，展现了其在教育技术领域的变革潜力。所有代码已开源：https://stevenshinechen.github.io/interactivesketchpad/。
原文摘要 · Abstract (English)
Humans have long relied on visual aids like sketches and diagrams to support reasoning and problem-solving. Visual tools, like auxiliary lines in geometry or graphs in calculus, are essential for understanding complex ideas. However, many tutoring systems remain text-based, providing feedback only through natural language. Leveraging recent advances in Large Multimodal Models (LMMs), this paper introduces Interactive Sketchpad, a tutoring system that combines language-based explanations with interactive visualizations to enhance learning. Built on a pre-trained LMM, Interactive Sketchpad is fine-tuned to provide step-by-step guidance in both text and visuals, enabling natural multimodal interaction with the student. Accurate and robust diagrams are generated by incorporating code execution into the reasoning process. User studies conducted on math problems such as geometry, calculus, and trigonometry demonstrate that Interactive Sketchpad leads to improved task comprehension, problem-solving accuracy, and engagement levels, highlighting its potential for transforming educational technologies. All code is available at: https://stevenshinechen.github.io/interactivesketchpad/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。