Python Engineer

Free Python and Machine Learning Tutorials

Become A Patron and get exclusive content! Get access to ML From Scratch notebooks, join a private Slack channel, get priority response, and more! I really appreciate the support!

back to course overview

Training Pipeline - PyTorch Beginner 06

28 Dec 2019

Learn all the basics you need to get started with this deep learning framework! In this part we improve the code from the last part and will learn how a complete training pipeline is implemented in PyTorch. We replace the manually computed loss and weight updates with a loss and an optimizer from the PyTorch framework, which can do the optimization for us. We will then see how a PyTorch model is implemented and used for the forward pass.

All code from this course can be found on GitHub.

Introduce Loss and Optimizer

# 1) Design model (input, output, forward pass with different layers) # 2) Construct loss and optimizer # 3) Training loop # - Forward = compute prediction and loss # - Backward = compute gradients # - Update weights import torch import torch.nn as nn # Linear regression # f = w * x # here : f = 2 * x # 0) Training samples X = torch.tensor([1, 2, 3, 4], dtype=torch.float32) Y = torch.tensor([2, 4, 6, 8], dtype=torch.float32) # 1) Design Model: Weights to optimize and forward function w = torch.tensor(0.0, dtype=torch.float32, requires_grad=True) def forward(x): return w * x print(f'Prediction before training: f(5) = {forward(5).item():.3f}') # 2) Define loss and optimizer learning_rate = 0.01 n_iters = 100 # callable function loss = nn.MSELoss() optimizer = torch.optim.SGD([w], lr=learning_rate) # 3) Training loop for epoch in range(n_iters): # predict = forward pass y_predicted = forward(X) # loss l = loss(Y, y_predicted) # calculate gradients = backward pass l.backward() # update weights optimizer.step() # zero the gradients after updating optimizer.zero_grad() if epoch % 10 == 0: print('epoch ', epoch+1, ': w = ', w, ' loss = ', l) print(f'Prediction after training: f(5) = {forward(5).item():.3f}')

Introduce the model class

# 1) Design model (input, output, forward pass with different layers) # 2) Construct loss and optimizer # 3) Training loop # - Forward = compute prediction and loss # - Backward = compute gradients # - Update weights import torch import torch.nn as nn # Linear regression # f = w * x # here : f = 2 * x # 0) Training samples, watch the shape! X = torch.tensor([[1], [2], [3], [4]], dtype=torch.float32) Y = torch.tensor([[2], [4], [6], [8]], dtype=torch.float32) n_samples, n_features = X.shape print(f'#samples: {n_samples}, #features: {n_features}') # 0) create a test sample X_test = torch.tensor([5], dtype=torch.float32) # 1) Design Model, the model has to implement the forward pass! # Here we can use a built-in model from PyTorch input_size = n_features output_size = n_features # we can call this model with samples X model = nn.Linear(input_size, output_size) ''' class LinearRegression(nn.Module): def __init__(self, input_dim, output_dim): super(LinearRegression, self).__init__() # define diferent layers self.lin = nn.Linear(input_dim, output_dim) def forward(self, x): return self.lin(x) model = LinearRegression(input_size, output_size) ''' print(f'Prediction before training: f(5) = {model(X_test).item():.3f}') # 2) Define loss and optimizer learning_rate = 0.01 n_iters = 100 loss = nn.MSELoss() optimizer = torch.optim.SGD(model.parameters(), lr=learning_rate) # 3) Training loop for epoch in range(n_iters): # predict = forward pass with our model y_predicted = model(X) # loss l = loss(Y, y_predicted) # calculate gradients = backward pass l.backward() # update weights optimizer.step() # zero the gradients after updating optimizer.zero_grad() if epoch % 10 == 0: [w, b] = model.parameters() # unpack parameters print('epoch ', epoch+1, ': w = ', w[0][0].item(), ' loss = ', l) print(f'Prediction after training: f(5) = {model(X_test).item():.3f}')