arXiv:2606.03951cs.CV2026-06中稿 · CVPR

将屏幕操作记录转化为可复用的图文教程,提升人与AI的学习效率。

Demo2Tutorial: From Human Experience to Multimodal Software Tutorials

论文配图:Demo2Tutorial: From Human Experience to Multimodal Software Tutorials
图 1 · 摘自论文原文
  • 通过多模态解析还原操作中的感知、动作与意图
  • 生成质量优于人工教程的结构化图文教程
  • 适合教学开发与GUI智能体训练使用

数字环境中的真实操作体验蕴含丰富流程知识,但长期未被有效利用。我们提出Demo2Tutorial框架,通过屏幕录制与交互日志收集人类操作,借助多模态动作解析器重建感知、动作与意图,并由步骤规划器抽象为层级任务图,最终由教程合成器生成结构化的图像-文本指令。在基于官方文档构建的新基准上评估显示,该框架生成的教程质量超越人工编写,显著优于基线方法。实验表明,其不仅能加速人类任务完成,还能显著提升下游GUI智能体的规划能力与泛化性能。代码与数据将开源。

原文摘要 · Abstract (English)

Human experience in digital environments offers a vast, underexplored resource of authentic, untrimmed interactions that contain rich procedural knowledge. We introduce Demo2Tutorial, a framework that transforms this experience captured via screen recordings and interaction logs into structured, multimodal software tutorials for teaching both humans and agents. Demo2Tutorial first collects human experience via a dedicated recorder, then parses raw experience using a multimodal Action Parser to reconstruct perception, action, and intent. A Step Planner then abstracts these steps into hierarchical task graphs representing goals and steps. Finally, a Tutorial Composer transforms the parsed experience into structured, reusable image-text instructions. We evaluate the tutorial generation quality on a new benchmark derived from official software documentation. We further demonstrate that this distilled representation benefits (i) human learning, by automatically generating multimodal tutorials, and (ii) agent learning, by improving downstream GUI-agent planning and generalization. Experiments show Demo2Tutorial produces high-quality tutorials that surpass human-authored ones and significantly outperform baseline methods, while enabling both faster human task completion and improved GUI agent planning, demonstrating that structured tutorials distilled from human experience can serve as effective knowledge representations for advancing both human learning and agent capabilities. Code and data will be available at https://github.com/showlab/Demo2Tutorial.

教程生成GUI智能体多模态解析知识蒸馏

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。