PyTorch Autograd

If tensors are the building blocks of PyTorch, then Autograd is the engine that enables neural networks to learn.

Autograd is PyTorch’s automatic differentiation system. It records every mathematical operation performed on tensors and automatically computes gradients using backpropagation.

Without Autograd, training modern deep learning models would require manually deriving and implementing derivatives for millions of parameters an impossible task for most real-world neural networks.

Why Do We Need Autograd? #

Training a neural network means adjusting parameters (weights and biases) so that the prediction error becomes smaller.

This adjustment depends on gradients.

Suppose we have$$y=x^2$$

The derivative is$$\frac{dy}{dx}=2x$$dxdy​=2x

Finding this derivative manually is trivial.

But a neural network is more like$$Loss=f(g(h(k(x))))$$Loss=f(g(h(k(x))))

where each layer depends on the previous one.

To compute$$\frac{\partial Loss}{\partial Weight}$$

we repeatedly apply the Chain Rule.

Doing this manually for millions of parameters is infeasible.

PyTorch solves this automatically.

---

How Autograd Works #

Autograd builds a Dynamic Computation Graph while your program executes.

Input Tensor
      │
      ▼
 Linear Layer
      │
      ▼
 Activation
      │
      ▼
 Loss

Forward Pass #

PyTorch records every operation.

x → y=x² → z=sin(y)

Instead of storing only values, PyTorch also remembers:

• operation performed
• parent tensors
• gradient functions

---

Backward Pass #

Calling

loss.backward()

makes PyTorch traverse the graph in reverse.

Loss
 ↑
Activation
 ↑
Linear Layer
 ↑
Input

The chain rule is applied automatically.

---

requires_grad=True #

Gradient tracking is disabled by default.

Enable it using

x = torch.tensor(3.0, requires_grad=True)

Now every operation involving x becomes part of the computation graph.

---

Example 1 — Computing a Simple Derivative #

import torch

x = torch.tensor(3.0, requires_grad=True)

y = x ** 2

y.backward()

print(x.grad)

Output

tensor(6.)

Why? #

Since$$y=x^2$$$$\frac{dy}{dx}=2x$$dxdy​=2x

At$$x=3$$x=3 $$2 \times 3 = 6$$

Autograd computed the derivative automatically.

---

Example 2 — Understanding the Chain Rule #

import torch

x = torch.tensor(4.0, requires_grad=True)

y = x ** 2

z = torch.sin(y)

z.backward()

print(x.grad)

Mathematically,$$z=\sin(x^2)$$

Using the chain rule,$$\frac{dz}{dx} = \cos(x^2)\times2x$$dxdz​=cos(x2)×2x

PyTorch produces exactly this result without requiring us to derive the equation manually.

This illustrates why Autograd is so powerful for deep neural networks composed of many nested operations.

---

Example 3 — Training a Single Neuron #

Suppose we want to predict whether a student gets placed based on CGPA.

Forward computation:

z = w*x + b

y_pred = torch.sigmoid(z)

loss = binary_cross_entropy_loss(y_pred, y)

Instead of deriving

• ∂Loss/∂Prediction
• ∂Prediction/∂z
• ∂z/∂w
• ∂z/∂b

manually,

we simply write

loss.backward()

print(w.grad)
print(b.grad)

PyTorch automatically computes

• gradient of weight
• gradient of bias

---

Working with Multiple Values #

Autograd also supports vectors and matrices.

Example from the notebook:

x = torch.tensor([1.0,2.0,3.0], requires_grad=True)

y = (x**2).mean()

y.backward()

print(x.grad)

Each element receives its own gradient.

tensor([...])

This is exactly how gradients are computed for batches of data during training.

---

Accessing Gradients #

Gradients are stored inside

tensor.grad

Example

print(x.grad)
print(w.grad)
print(b.grad)

The .grad attribute becomes available only after calling

backward()

---

Gradient Accumulation #

An important PyTorch behavior is that gradients accumulate.

Example:

y.backward()

print(x.grad)

x.grad.zero_()

Without clearing gradients, every subsequent call to backward() adds new gradients to the existing ones.

During training, optimizers therefore perform

optimizer.zero_grad()

loss.backward()

optimizer.step()

every iteration.

---

Disabling Gradient Tracking #

Gradients are needed only during training.

During inference, gradient computation wastes memory.

Your notebook demonstrates three common approaches.

1. requires_grad_(False) #

x.requires_grad_(False)

---

2. detach() #

z = x.detach()

The detached tensor shares data but no longer belongs to the computation graph.

---

3. torch.no_grad() #

with torch.no_grad():
    prediction = model(x)

This is the preferred approach during inference because it reduces memory usage and improves execution speed.

---

Complete Training Workflow #

A simplified training iteration looks like this:

# Forward Pass
prediction = model(x)

# Compute Loss
loss = criterion(prediction, target)

# Clear Previous Gradients
optimizer.zero_grad()

# Backpropagation
loss.backward()

# Update Parameters
optimizer.step()

This sequence is repeated for every batch throughout training.

---

Best Practices #

✔ Enable requires_grad=True only for trainable parameters.

✔ Always clear gradients before backpropagation.

✔ Use torch.no_grad() during inference.

✔ Inspect .grad for debugging and understanding optimization behavior.

✔ Remember that PyTorch builds computation graphs dynamically during execution.

---

Key Takeaways #

• Autograd is PyTorch’s automatic differentiation engine.
• It constructs a dynamic computation graph during the forward pass.
• Calling backward() automatically applies the chain rule to compute gradients.
• Computed gradients are stored in the .grad attribute.
• Gradients accumulate unless they are explicitly cleared.
• Disable gradient tracking during inference using torch.no_grad() or .detach().
• The examples in this notebook—from simple derivatives to training a perceptron—demonstrate the exact workflow used in real-world deep learning models.

PyTorch Autograd Quiz #