让本科生通过自己生成的数据理解强化学习的每一步计算过程。
Q-Learning Lab: Teaching Reinforcement Learning Through Learner-Generated Trace Analysis
- 提供实时展示Q值更新的交互界面，可见每步计算细节。
- 可导出完整训练日志，支持学生自主分析学习曲线与策略变化。
- 适合初学者在无安装环境下动手实践，培养反思性学习能力。
强化学习通常通过贝尔曼方程引入，但该公式对本科生而言仍显抽象：他们看到策略箭头收敛，却很少观察每个价值如何计算或为何选择某个动作。我们提出Q-Learning Lab，一个单文件、基于浏览器、双语（泰语/英语）的工具，用于教学表格型Q-learning，无需安装。除了常见的网格世界可视化——5×5世界中颜色编码的Q值和策略箭头——该工具还提供一个实时贝尔曼替换面板，显示每一步的数值更新，并将每次转移记录为可导出的追踪日志，包括动作前的完整Q行、ε-贪婪探索下的贪心与随机决策、以及撞墙事件。核心贡献是“学习-导出-分析”循环：学习者运行自己的智能体，导出完整追踪日志为CSV，自行分析生成学习曲线、价值热图和访问地图，将被动演示转化为自我生成数据源，促进反思性探究。我们通过三项互补评估验证该工具：(i) 在相同马尔可夫决策过程上，学习到的价值与策略与值迭代基准一致；(ii) 对α、γ、ε进行超参数扫描，证明工具中的每个教学主张均可复现；(iii) 通过奖励编辑实验，利用最优策略作为基准，区分两种行为相同但诊断意义相反的失败模式——探索失败与真正的奖励误设。我们还对比现有网格世界可视化工具，阐述其基于“做中学”的教学理念，并附有50分钟课程计划。工具及所有实验代码均开源。
原文摘要 · Abstract (English)
Reinforcement learning is usually introduced through the Bellman update, yet the equation often remains abstract to undergraduates: they watch policy arrows converge but rarely observe how each value is computed or why an action is chosen. We present Q-Learning Lab, a single-file, browser-based, bilingual (Thai/English) tool for teaching tabular Q-learning that requires no installation. Beyond the usual gridworld visualization - color-coded Q-values and policy arrows on a $5 \times 5$ world - the tool exposes a live Bellman-substitution panel showing the numeric update at every step, and logs each transition, including the full pre-action Q-row, the greedy-versus-random decision under $\varepsilon$-greedy exploration, and wall-collision events, into an exportable trace. The central contribution is a learn-export-analyze loop: learners run their own agent, export the complete trace as CSV, and analyze it themselves, producing learning curves, value heatmaps, and visitation maps, turning a passive demonstration into a source of learner-generated data for reflective inquiry. We validate the tool without human-subject data through three complementary evaluations: (i) correctness of the learned values and policy against a value-iteration ground truth on the identical MDP; (ii) hyperparameter sweeps over $α$, $γ$, and $\varepsilon$ showing that every pedagogical claim the tool makes is reproducible; and (iii) a reward-editing study that uses the ground-truth optimal policy to separate two behaviorally identical but diagnostically opposite failure modes - an exploration failure versus genuine reward misspecification - that a single edited reward can produce. We also compare the tool against existing gridworld visualizers, describe its grounding in learning-by-doing pedagogy, and include a 50-minute lesson plan. The tool and all experiment code are openly available.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。