Gradient Descent and Backpropagation
Gradient descent steps a model's weights against the loss gradient; backpropagation computes that gradient with the chain rule. Run in NumPy: a learning rate of 1.1 diverged, and a net learned XOR.
Gradient descent trains a model by repeatedly stepping its weights against the loss gradient, w = w - lr * dLoss/dw. Backpropagation computes that gradient for every weight by applying the chain rule from the output layer backward. Interviewers expect the learning rate trade-off: too small converges slowly, too large overshoots and can make the loss grow without bound.
Gradient descent trains a model by repeatedly moving its weights a small step in the direction that lowers the loss. Backpropagation is the method for computing the gradient of the loss with respect to every weight in a network, by applying the chain rule from the output backward.
- The gradient descent update rule is w = w - lr * dLoss/dw, applied to every weight, where the learning rate lr sets the step size.
- Fitting y = 3x + 2 with 100 steps of gradient descent, lr 0.1 reached w = 2.986 and b = 1.992, lr 0.01 stopped at w = 1.639, and lr 1.1 diverged to an MSE near 1e18.
- Backpropagation computes the gradient of the loss for every weight with the chain rule, and one backward pass costs roughly as much as one forward pass.
- Common local derivatives in backpropagation are 1 - tanh squared for tanh, s(1 - s) for sigmoid, and 1 or 0 for ReLU; sigmoid with cross-entropy gives p - y.
- A gradient check compares analytic gradients with finite differences; for a NumPy XOR network the two agreed with a relative error of 8.5e-11.
- Mini-batch gradient descent uses 32 to a few hundred examples per step and is what almost every deep learning library means by SGD.
01The idea: walk downhill on the loss
A model has weights. A loss function scores how wrong the model's predictions are on the training data. Training means finding weights that make the loss small. The gradient of the loss with respect to a weight tells you how the loss changes if you nudge that weight, and it points uphill, so you step the opposite way:
w = w - lr * dLoss/dw
The learning rate lr is the step size. It is the single most important setting. Too small and training crawls, too large and you overshoot the valley and bounce out of it. For linear regression with mean squared error the loss surface is a bowl with one minimum, so descent is guaranteed to find it with a suitable step. For neural networks the surface has many dips and flat regions, and descent finds a good dip rather than a provably best one.
02Gradient descent on a line, with four learning rates
The data is 200 points from y = 3x + 2 plus small noise. The model is y = wx + b with mean squared error loss, so the gradients are 2mean(errx) for w and 2mean(err) for b. Each run starts from w = b = 0 and takes 100 steps.
import numpy as np
rng = np.random.default_rng(0)
x = rng.uniform(-1, 1, 200)
y = 3.0 * x + 2.0 + rng.normal(0, 0.1, 200) # true w=3, b=2
def fit(lr, steps):
w = b = 0.0
for _ in range(steps):
err = (w * x + b) - y
gw, gb = 2 * np.mean(err * x), 2 * np.mean(err) # d(MSE)/dw, d(MSE)/db
w -= lr * gw; b -= lr * gb
return w, b, np.mean(((w * x + b) - y) ** 2)
for lr in (0.01, 0.1, 0.9, 1.1):
w, b, loss = fit(lr, 100)
print(f"lr={lr:<4} w={w:9.3f} b={b:9.3f} mse={loss:.4g}")
A = np.c_[x, np.ones_like(x)]
print("closed form:", np.linalg.lstsq(A, y, rcond=None)[0].round(3))Output:
lr=0.01 w= 1.639 b= 1.852 mse=0.7321
lr=0.1 w= 2.986 b= 1.992 mse=0.01042
lr=0.9 w= 2.987 b= 1.992 mse=0.01042
lr=1.1 w=-141810354.639 b=-1145580091.932 mse=1.346e+18
closed form: [2.987 1.992]The closed-form least squares answer, 2.987 and 1.992, is where descent should land, and with lr 0.1 and 0.9 it did to three decimals. At 0.01 the weights were still moving after 100 steps. At 1.1 each step overshoots the minimum by more than it started away from, so the error grows every iteration and ends near 1e18. The true values are 3 and 2, and the noise explains why the best fit is 2.987 and 1.992. The stability limit here depends on the data scale; do not take 1.0 as a universal threshold.
03Batch, stochastic and mini-batch
The code above computes the gradient from all 200 examples before each step, which is batch gradient descent. It is exact but costs a full pass per step, impractical with millions of examples.
Stochastic gradient descent (SGD) updates after each single example. Steps are noisy but cheap, and the noise can help escape poor regions. Mini-batch descent uses 32 to a few hundred examples per step, uses vector hardware well, and is what almost every deep learning library means by SGD today. Variants like momentum and Adam change how the step is built from past gradients; they still compute gradients the same way, which is where backpropagation comes in.
04Backpropagation: the chain rule, applied backward
A network is a chain of simple functions: multiply by weights, add bias, apply an activation, repeat, then compute a loss. The chain rule says the derivative of a composition is the product of the derivatives of its parts. Backpropagation evaluates that product efficiently.
First do a forward pass, storing each layer's output. Then, starting at the loss, compute how much the loss changes with respect to the output layer's input. For a sigmoid output with cross-entropy loss this simplifies neatly to prediction minus label. Multiply by the previous layer's output to get the weight gradient, and multiply by the weights to pass the signal further back, scaling by each activation's derivative on the way. One backward pass costs roughly the same as one forward pass, which is why training deep networks is feasible at all. Doing the derivatives separately for each weight would cost far more.
The local derivatives you must remember: tanh gives 1 - tanh squared, sigmoid gives s(1-s), ReLU gives 1 for positive input and 0 otherwise.
05A two-layer network learns XOR, with a gradient check
XOR cannot be learned by a straight line, so it needs the hidden layer. The code below has a tanh hidden layer of 4 units, a sigmoid output, and the backward pass written out by hand with NumPy. Before training it compares one analytic gradient against a numerical estimate by nudging the weight by plus and minus 1e-5. This gradient check is the standard way to catch backpropagation bugs.
import numpy as np
rng = np.random.default_rng(1)
X = np.array([[0,0],[0,1],[1,0],[1,1]], float)
Y = np.array([[0],[1],[1],[0]], float) # XOR: not linearly separable
W1 = rng.normal(0, 1, (2, 4)); b1 = np.zeros(4)
W2 = rng.normal(0, 1, (4, 1)); b2 = np.zeros(1)
sig = lambda z: 1 / (1 + np.exp(-z))
def forward(W1, b1, W2, b2):
h = np.tanh(X @ W1 + b1)
p = sig(h @ W2 + b2)
return h, p
def loss_fn(W1, b1, W2, b2):
_, p = forward(W1, b1, W2, b2)
return -np.mean(Y*np.log(p) + (1-Y)*np.log(1-p))
def backward(W1, b1, W2, b2):
h, p = forward(W1, b1, W2, b2)
dz2 = (p - Y) / len(X) # sigmoid + cross-entropy collapses to p - y
dW2 = h.T @ dz2; db2 = dz2.sum(0)
dh = dz2 @ W2.T
dz1 = dh * (1 - h**2) # tanh' = 1 - tanh^2
dW1 = X.T @ dz1; db1 = dz1.sum(0)
return dW1, db1, dW2, db2
# gradient check on one weight
g = backward(W1, b1, W2, b2)[0][0, 0]
eps = 1e-5
Wp = W1.copy(); Wp[0, 0] += eps; Wm = W1.copy(); Wm[0, 0] -= eps
num = (loss_fn(Wp, b1, W2, b2) - loss_fn(Wm, b1, W2, b2)) / (2 * eps)
print(f"analytic {g:.8f} numeric {num:.8f} rel err {abs(g-num)/abs(g+num):.1e}")
print("initial loss", round(loss_fn(W1, b1, W2, b2), 4))
for step in range(1, 2001):
dW1, db1, dW2, db2 = backward(W1, b1, W2, b2)
W1 -= 0.5*dW1; b1 -= 0.5*db1; W2 -= 0.5*dW2; b2 -= 0.5*db2
if step in (100, 500, 2000):
print("step", step, "loss", round(loss_fn(W1, b1, W2, b2), 4))
print("predictions", forward(W1, b1, W2, b2)[1].round(3).ravel())Output:
analytic -0.02928259 numeric -0.02928259 rel err 8.5e-11
initial loss 0.6712
step 100 loss 0.1231
step 500 loss 0.0114
step 2000 loss 0.0024
predictions [0. 0.997 0.997 0.004]The two gradients agree to eight decimals, so the backward pass is right. The loss fell steadily and the four predictions are about 0, 1, 1, 0, which is XOR. This ran with one fixed random seed. Different seeds can land in different local solutions and need more steps, so a single run does not show it always works.
06What goes wrong in practice
Learning rate: run a quick sweep like the one above, or use a schedule that starts higher and decays. Vanishing gradients: multiplying many derivatives below 1, as sigmoid and tanh produce, shrinks the signal toward zero in early layers; ReLU activations, careful initialisation and residual connections reduce this. Exploding gradients: the same product above 1 blows up; gradient clipping caps it. Unscaled features: if one input ranges over thousands and another over fractions, the loss surface is a long narrow trough and descent zigzags; standardise inputs first.
For tabular problems and interviews, you will rarely write backpropagation yourself. Libraries like PyTorch do automatic differentiation. What matters is being able to explain what the gradient means and why the backward pass works.
07Common interview questions
Q: Write the gradient descent update rule.
A: w = w - lr * dLoss/dw, applied to every weight, where lr is the learning rate.
Q: What happens if the learning rate is too large?
A: Steps overshoot the minimum and the loss can oscillate or grow without bound. In the run above, lr 1.1 sent the error to about 1e18.
Q: What does backpropagation compute?
A: The gradient of the loss with respect to every weight, using the chain rule from the output layer back to the input.
Q: Difference between batch, stochastic and mini-batch descent?
A: They use all examples, one example or a small group per step; mini-batch balances gradient noise against computation and is the usual choice.
Q: What is the vanishing gradient problem?
A: In deep networks, repeated multiplication by derivatives smaller than 1 makes early layers' gradients tiny, so they barely learn.
Q: How do you verify a backpropagation implementation?
A: Compare its gradients to finite-difference numerical gradients; they should agree to many decimals.