用纯Python手写框架，让学生真正理解机器学习系统的底层运行机制。
TinyTorch: Building Machine Learning Systems from First Principles
- 从零构建张量、自动求导、优化器等核心模块，代码可读可调试。
- 通过67年关键突破的复现，实现从感知机到Transformer的系统级训练。
- 仅需4GB内存无GPU，适合教学与初学者掌握ML系统工程思维。
机器学习教育存在算法与系统之间的根本鸿沟：学生学习梯度下降却不知内存开销，研究注意力机制却不分析O(N²)扩展性，理解优化器理论却不清楚Adam为何需要3倍于SGD的内存。这种‘算法-系统断裂’导致从业者能训练模型却无法调试内存问题、优化推理延迟或权衡部署方案——而这正是产业界急需的‘机器学习系统工程’能力。本文提出TinyTorch，一个由20个模块组成的课程体系，采用‘基于实现的系统教学法’：学生在纯Python中逐步构建PyTorch的核心组件（张量、autograd、优化器、CNN、Transformer），最终形成一个完整可运行的框架，所有操作均由自己编写的代码实现。设计遵循三大原则：渐进式复杂度披露、从第一模块即集成性能剖析、以里程碑式重构67年来的关键进展——从1958年感知机到2017年Transformer，再到符合MLPerf标准的基准测试。整个课程仅需4GB内存且无需GPU，证明深度机器学习系统理解无需专用硬件即可实现。课程已开源，地址为mlsysbook.ai/tinytorch。
原文摘要 · Abstract (English)
Machine learning education faces a fundamental gap: students learn algorithms without understanding the systems that execute them. They study gradient descent without measuring memory, attention mechanisms without analyzing O(N^2) scaling, optimizer theory without knowing why Adam requires 3x the memory of SGD. This "algorithm-systems divide" produces practitioners who can train models but cannot debug memory failures, optimize inference latency, or reason about deployment trade-offs--the very skills industry demands as "ML systems engineering." We present TinyTorch, a 20-module curriculum that closes this gap through "implementation-based systems pedagogy": students construct PyTorch's core components (tensors, autograd, optimizers, CNNs, transformers) in pure Python, building a complete framework where every operation they invoke is code they wrote. The design employs three patterns: "progressive disclosure" of complexity, "systems-first integration" of profiling from the first module, and "build-to-validate milestones" recreating 67 years of ML breakthroughs--from Perceptron (1958) through Transformers (2017) to MLPerf-style benchmarking. Requiring only 4GB RAM and no GPU, TinyTorch demonstrates that deep ML systems understanding is achievable without specialized hardware. The curriculum is available open-source at mlsysbook.ai/tinytorch.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。