Lecture 5: Deep Learning Overview

(Last updated: Jan 27, 2026)

This course introduces deep learning techniques, such as deep neural networks, loss functions, and gradient descent.

Check the GenAI usage policy if you are using the course materials with GenAI for self-study and fact-checking.

Preparation

Read the required course readings.

Lecture

Below are the slides:

• Slides for Lecture 5: Deep Learning Overview

Required Course Readings

• Section 2.4.4 (Neural Networks, including 2.4.4.1 and 2.4.4.2) in the ML4Design lecture notes (Bozzon, 2023)
• Section 10.7 (Fitting a Neural Network, including 10.7.1, 10.7.2, 10.7.3, and 10.7.4) in book An Introduction to Statistical Learning (James et al., 2013). Do not worry about the chain rule math in 10.7.1 (focus on understanding why we need backpropagation).

Optional Course Readings

• Section 5.2 (Capacity, Overfitting and Underfitting, including 5.2.1, and 5.2.2) in book Deep Learning (Goodfellow et al., 2016).
• Section 6.2 (Shrinkage Methods, including 6.2.1, 6.2.2, and 6.2.3) in book An Introduction to Statistical Learning (James et al., 2013)
• Section 7.4 (Dataset Augmentation) in book Deep Learning (Goodfellow et al., 2016).

Additional Resources

Below is a well-known paper that gives a nice overview of deep learning:

• LeCun, Y., Bengio, Y., & Hinton, G. (2015). Deep learning. Nature.

Below is a collection of projects by OpenAI (which creates ChatGPT):

• OpenAI Research Projects in Deep Learning

Below is a collection of project by DeepMind (which creates AlphaGo):

• DeepMind Research Projects in Deep Learning

Below is a collection of AI experiments by Google:

• Google AI Experiments