🎤 PyTorch Karaoke, or How to Stop Forgetting optimizer.zero_grad()

Many DL beginners break not on complex network architectures, but on the mundane PyTorch boilerplate. The order of calls in the training loop is something you have to Google over and over until it's etched into your subconscious.

Forgot to set the model to train()? Get incorrect weights due to Dropout/BatchNorm.
Forgot zero_grad()? Congratulations, gradients accumulate, training goes down the drain.
Put step() before backward()? Well, you get it.

Stumbled upon a video that evokes two feelings at once: extreme cringe and respect.
A guy in his underwear with a microphone in the middle of a mess just took the 5 steps of Backpropagation and set them to a tune that you can't get out of your head.
It seems to work better than 10 hours of boring lectures from experts reading off slides.

For those who are out of the loop, let me remind you of the only correct order of actions that you should tattoo on your subconscious (or learn the song):

1️⃣ model.train() — set the model to training mode.
2️⃣ y_pred = model(x) — Forward pass.
3️⃣ loss = loss_fn(y_pred, y) — Calculate how wrong we are.
4️⃣ optimizer.zero_grad() — Zero out gradients before the next step, otherwise they accumulate and you'll fly into space.
5️⃣ loss.backward() — Compute gradients (backpropagation).
6️⃣ optimizer.step() — Take an optimizer step.

If only there were a version for TensorFlow, but I'm afraid you'd have to write a three-act opera just to initialize variables 🌚

#for_fun