Week 6 · Lesson 3 of 8
Gradient descent tuning
Gradient descent tuning
Learn how learning rate, batch size and schedules shape convergence. We'll compare SGD, momentum and Adam on the same loss surface and build intuition for picking hyperparameters.
Duration
19 min
Watched
38%
Completion rule
Watch 90%
Chapters
03:41
Learning rate too high → loss oscillates. Try 1e-3 then scale.
12:04
Warm-up + cosine schedule is the default for transformers.
Your AI tutor has watched this lesson
It answers from the transcript and course readings, and links back to the moment in the video.
Lecture slides — Week 6.pdf
3.1 MB
gradient_descent_lab.ipynb
Open in JupyterHub
Distill: Why momentum really works
distill.pub
ML
Marcus Lee 05:20
Why does Adam sometimes generalise worse than SGD?
Teacher answered14 upvotes
PS
Priya Sharma 09:02
Is batch size 1 the same as pure SGD?
3 replies