TinyTorch: Building Machine Learning Systems from First Principles

Machine learning education faces a fundamental gap: students learn algorithms without understanding the systems that execute them. They study gradient descent without measuring memory, attention mechanisms without analyzing O(N^2) scaling, optimizer theory without knowing why Adam requires 3x the memory of SGD. This “algorithm-systems divide” produces practitioners who can train models but cannot debug memory failures, optimize inference latency, or reason about deployment trade-offs–the very skills industry demands as “ML systems engineering.” We present TinyTorch, a 20-module curriculum that closes this gap through “implementation-based systems pedagogy”: students construct PyTorch’s core components (tensors, autograd, optimizers, CNNs, transformers) in pure Python, building a complete framework where every operation they invoke is code they wrote. The design employs three patterns: “progressive disclosure” of complexity, “systems-first integration” of profiling from the first module, and “build-to-validate milestones” recreating 67 years of ML breakthroughs–from Perceptron (1958) through Transformers (2017) to MLPerf-style benchmarking. Requiring only 4GB RAM and no GPU, TinyTorch demonstrates that deep ML systems understanding is achievable without specialized hardware. The curriculum is available open-source at mlsysbook.ai/tinytorch.

💡 Research Summary

The paper identifies a critical gap in contemporary machine‑learning education: students are taught algorithms without a deep understanding of the systems that execute them. This “algorithm‑systems divide” leaves graduates able to train models but unable to diagnose memory overflows, optimize inference latency, or make informed deployment trade‑offs—skills that industry now demands for ML systems engineers. To bridge this divide, the authors introduce TinyTorch, a 20‑module, implementation‑first curriculum that guides learners to rebuild the core of PyTorch from scratch using pure Python.

Each module adds a concrete piece of functionality—tensors, activation functions, layers, loss functions, data loading, autograd, optimizers, training loops, CNNs, tokenizers, embeddings, attention, transformers, profiling, quantization, compression, acceleration, memoization, benchmarking, and finally a capstone project. The design follows three pedagogical patterns. First, progressive disclosure gradually reveals complexity: the tensor engine is introduced early, but gradient tracking is only activated in a later module, preventing cognitive overload while preserving a unified mental model. Second, systems‑first integration embeds memory and compute profiling from the very first module, so students see that Adam’s two state buffers double memory usage, that O(N²) attention scales quadratically, and that techniques such as gradient accumulation or activation checkpointing have measurable effects. Third, a build‑to‑validate approach requires learners to reproduce historic milestones—from the 1958 perceptron through LeNet, AlexNet, ResNet, and finally modern transformers—using only the code they have written, providing concrete correctness criteria and performance targets.

The curriculum is deliberately lightweight: it runs on a machine with 4 GB of RAM and no GPU, demonstrating that deep systems insight does not require specialized hardware. Automated unit tests and profiling hooks give immediate feedback, and the final modules culminate in an MLPerf‑style benchmark where students compare their TinyTorch implementation against published baselines.

In the related‑work discussion, TinyTorch is positioned against micrograd (scalar‑only autograd), MiniTorch (tensor library with optional GPU), and tinygrad (full‑stack compiler‑oriented framework). While those projects illuminate specific aspects, none combine a step‑wise scaffolding, systematic profiling, and a historical validation pathway. TinyTorch therefore fills a unique niche: it trains “framework engineers” rather than merely “framework users.”

The authors outline an empirical evaluation plan involving pre‑ and post‑course surveys, code‑quality metrics, profiling data, and industry interviews. Expected outcomes include improved systems‑thinking, debugging proficiency, and readiness for production‑scale ML engineering roles, directly addressing the talent shortage highlighted by recent industry surveys.

In summary, TinyTorch embodies the philosophy that to truly understand a system, one must build it. By coupling algorithmic fundamentals with hands‑on systems construction, the curriculum equips learners with the mental models and practical skills needed to scale modern AI models efficiently, thereby narrowing the algorithm‑systems divide and preparing the next generation of ML systems engineers.