Python Engineer

Free Python and Machine Learning Tutorials

Become A Patron and get exclusive content! Get access to ML From Scratch notebooks, join a private Slack channel, get priority response, and more! I really appreciate the support!

back to course overview

Gradient Descent Using Autograd - PyTorch Beginner 05

27 Dec 2019

Learn all the basics you need to get started with this deep learning framework! In this part we will learn how we can use the autograd engine in practice. First we will implement Linear regression from scratch, and then we will learn how PyTorch can do the gradient calculation for us.

All code from this course can be found on GitHub.

Backpropagation Manually

import numpy as np # Compute every step manually # Linear regression # f = w * x # here : f = 2 * x X = np.array([1, 2, 3, 4], dtype=np.float32) Y = np.array([2, 4, 6, 8], dtype=np.float32) w = 0.0 # model output def forward(x): return w * x # loss = MSE def loss(y, y_pred): return ((y_pred - y)**2).mean() # J = MSE = 1/N * (w*x - y)**2 # dJ/dw = 1/N * 2x(w*x - y) def gradient(x, y, y_pred): return np.dot(2*x, y_pred - y).mean() print(f'Prediction before training: f(5) = {forward(5):.3f}') # Training learning_rate = 0.01 n_iters = 20 for epoch in range(n_iters): # predict = forward pass y_pred = forward(X) # loss l = loss(Y, y_pred) # calculate gradients dw = gradient(X, Y, y_pred) # update weights w -= learning_rate * dw if epoch % 2 == 0: print(f'epoch {epoch+1}: w = {w:.3f}, loss = {l:.8f}') print(f'Prediction after training: f(5) = {forward(5):.3f}')

Backpropagation with Autograd

import torch # Here we replace the manually computed gradient with autograd # Linear regression # f = w * x # here : f = 2 * x X = torch.tensor([1, 2, 3, 4], dtype=torch.float32) Y = torch.tensor([2, 4, 6, 8], dtype=torch.float32) w = torch.tensor(0.0, dtype=torch.float32, requires_grad=True) # model output def forward(x): return w * x # loss = MSE def loss(y, y_pred): return ((y_pred - y)**2).mean() print(f'Prediction before training: f(5) = {forward(5).item():.3f}') # Training learning_rate = 0.01 n_iters = 100 for epoch in range(n_iters): # predict = forward pass y_pred = forward(X) # loss l = loss(Y, y_pred) # calculate gradients = backward pass l.backward() # update weights #w.data = w.data - learning_rate * w.grad with torch.no_grad(): w -= learning_rate * w.grad # zero the gradients after updating w.grad.zero_() if epoch % 10 == 0: print(f'epoch {epoch+1}: w = {w.item():.3f}, loss = {l.item():.8f}') print(f'Prediction after training: f(5) = {forward(5).item():.3f}')