# Autograd - PyTorch Beginner 03

Learn all the basics you need to get started with this deep learning framework! In this part we learn how to calculate gradients using the autograd package in PyTorch. This tutorial contains the following topics:

`requires_grad`

attribute for Tensors- Computational graph
- Backpropagation (brief explanation)
- How to stop autograd from tracking history
- How to zero (empty) gradients

All code from this course can be found on GitHub.

## The Autograd package

The autograd package provides automatic differentiation for all operations on Tensors. To tell PyTorch that we want the gradient, we have to set `requires_grad=True`

. With this attribute set, all operations on the tensor are tracked in the computational graph.

```
import torch
# requires_grad = True -> tracks all operations on the tensor.
x = torch.randn(3, requires_grad=True)
y = x + 2
# y was created as a result of an operation, so it has a grad_fn attribute.
# grad_fn: references a Function that has created the Tensor
print(x) # created by the user -> grad_fn is None
print(y)
print(y.grad_fn)
# Do more operations on y
z = y * y * 3
print(z)
z = z.mean()
print(z)
```

## Let's compute the gradients with backpropagation

When we finish our computation we can simply call `.backward()`

and have all the gradients computed automatically. The gradient for this tensor will be accumulated into `.grad`

attribute. It is the partial derivate of the function w.r.t. the tensor.

```
z.backward()
print(x.grad) # dz/dx
```

Generally speaking, `torch.autograd`

is an engine for computing vector-Jacobian product. It computes partial derivates while applying the chain rule.

```
# Model with non-scalar output:
# If a Tensor is non-scalar (more than 1 elements), we need to specify arguments for backward()
# specify a gradient argument that is a tensor of matching shape.
# needed for vector-Jacobian product
x = torch.randn(3, requires_grad=True)
y = x * 2
for _ in range(10):
y = y * 2
print(y)
print(y.shape)
v = torch.tensor([0.1, 1.0, 0.0001], dtype=torch.float32)
y.backward(v)
print(x.grad)
```

## Stop a tensor from tracking history

For example during our training loop when we want to update our weights then this update operation should not be part of the gradient computation. We have 3 options to stop gradient calculations:

`x.requires_grad_(False)`

`x.detach()`

- wrap in
`with torch.no_grad():`

`.requires_grad_(...)`

changes an existing flag in-place:

```
a = torch.randn(2, 2)
print(a.requires_grad)
b = ((a * 3) / (a - 1))
print(b.grad_fn)
a.requires_grad_(True)
print(a.requires_grad)
b = (a * a).sum()
print(b.grad_fn)
```

`.detach()`

: get a new Tensor with the same content but no gradient computation:

```
a = torch.randn(2, 2, requires_grad=True)
print(a.requires_grad)
b = a.detach()
print(b.requires_grad)
```

### wrap in `with torch.no_grad()`

:

```
a = torch.randn(2, 2, requires_grad=True)
print(a.requires_grad)
with torch.no_grad():
print((x ** 2).requires_grad)
```

## Empty gradients!

`backward()`

**accumulates the gradient** for this tensor into the `.grad`

attribute. We need to be careful during optimization !!!

-> Use `.zero_()`

to empty the gradients before a new optimization step!

```
weights = torch.ones(4, requires_grad=True)
for epoch in range(3):
# just a dummy example
model_output = (weights*3).sum()
model_output.backward()
print(weights.grad)
# optimize model, i.e. adjust weights...
with torch.no_grad():
weights -= 0.1 * weights.grad
# this is important! It affects the final weights & output
weights.grad.zero_()
print(weights)
print(model_output)
```

### Optimizer has `zero_grad()`

method

(We will learn about optimizer in tutorial #6)

```
optimizer = torch.optim.SGD([weights], lr=0.1)
# During training:
optimizer.step()
optimizer.zero_grad()
```

**Join My Newsletter! **Get Python and ML tips emailed directly to your inbox. Each month you’ll get a summary of all the content I created, including the newest videos, articles, promotions, tips, and more.

# TensorFlow 2 Beginner

Learn all the necessary basics to get started with TensorFlow 2 and Keras.

# PyTorch Beginner

Learn all the necessary basics to get started with this deep learning framework.

# ML From Scratch

Implement popular Machine Learning algorithms from scratch using only built-in Python modules and numpy.