In this course, you will learn the fundamental principles of deep learning by building your very own deep learning library, similar to PyTorch, completely from scratch. This course includes all three courses in the Flamethrower Core series, for 33% off the price of buying all three separately. You'll start by learning how automatic differentiation works, and you'll build your own automatic differentiation engine based on the popular autograd library. From there, you'll be introduced to different neural network layer types, regularization strategies, loss functions, and more. As each of these concepts are introduced, you'll implement them in your library and test them out on real world data. We'll integrate material from statistics, numerical analysis, and Bayesian statistics to give you the most well rounded perspective of deep learning available in any course. By the time the course is finished, you'll have a solid grounding in both the theoretical and applied aspects of the field, and you'll have your own deep learning library to apply to problems of your own creation. View the entire course syllabus below, along with preview lessons. Be sure to click the drop down arrow to see the syllabus in its entirety.
Course Curriculum
- What Automatic Differentiation Isn't (25:12)
- The Types of Automatic Differentiation Systems (20:25)
- Computational Graphs, Compositions of Primitives, and Reverse Mode Differentation
- Forward Mode Differentiation and Dual Numbers (6:14)
- Nodes: Building a Computational Graph (42:48)
- Variables (27:34)
- Tensors (39:26)
- Creating Callable Primitives and Tracing a Computation (56:19)
- Creating a Tensor Library (24:13)
- An Introduction to Numpy Broadcasting (16:27)
- Differentiating Primitives (25:43)
- Validating Gradients via Gradient Checking (41:04)
- Implementing a Topological Sort - Kahn's Algorithm (16:19)
- Implementing a Topological Sort - Depth First Search (23:42)
- Implementing a Topological Sort - Faster Depth First Search (10:31)
- Understanding Backpropagation (22:37)
- Implementing Backpropagation (37:31)
- The History of Neural Networks
- The Module Base Class (19:54)
- Practicalities: Using Logging
- Parameter Initialization Strategies - Xavier Initialization (19:13)
- Parameter Initialization Strategies - Glorot Uniform Initialization (6:10)
- Parameter Initialization Strategies - The Implementation (11:24)
- The Linear Layer (11:41)
- Regression and Classification (10:35)
- Activation Functions (17:35)
- Statistical Estimators, Underfitting, and Overfitting (23:28)
- Regularization for Better Generalization (25:48)
- Regularization - Label Smoothing (29:16)
- Regularization - The Elastic Net (8:20)
- Implementing Regularization (7:04)
- A Drop of Dropout - Part 1 (4:04)
- A Drop of Dropout - Part 2 (22:50)
- Implementing Dropout (8:38)
- A Batch of Batch Norm (18:08)
- Implementing Batch Norm (6:28)
- An Introduction to Neural Network Optimization (21:04)
- The Optimizer Base Class (13:08)
- Loss Functions - Introduction (7:37)
- Loss Functions - Maximum Likelihood Estimation, KL Divergence, Wasserstein Metrics (14:09)
- Loss Functions - Deriving Mean Squared Error (17:05)
- Loss Functions - Deriving Cross Entropy Loss (13:24)
- Loss Functions - Maximum A Posteriori (13:24)
- Loss Functions - Maximum a Posteriori with a Gaussian Prior (7:38)
- Loss Functions - Maximum a Posteriori with a Laplacian Prior (9:14)
- Loss Functions - The Implementation (14:35)
- Gradient Descent - The Theory (41:01)
- Improving Gradient Descent - Momentum (13:19)
- Improving Gradient Descent - Nesterov Momentum (5:58)
- Gradient Descent - The Implementation (11:15)
- Scheduling Learning Rates (33:34)
- Hyperparameter Search (12:53)