Stanford CS230 | Autumn 2025 | Lecture 3: Full Cycle of a DL project

This lecture teaches the core ideas behind supervised learning for regression, focusing on the classic and foundational method: linear regression. In supervised learning, you are given input–output pairs: each input x is a vector of features (for example, house characteristics like square footage, number of bedrooms, and location), and each output y is the real number you want to predict (such as the house price). The goal is to learn a function f that maps any new input x to a good prediction of y. Linear regression assumes this mapping is a straight line (technically, a hyperplane) in feature space: f(x) = w^T x + b, where w is a vector of weights and b is a bias (intercept).

To measure how well the model predicts, the lecture uses Mean Squared Error (MSE), which averages the squares of the differences between actual values and predicted values. Squaring emphasizes larger mistakes more than smaller ones, which helps the optimization focus on big errors. The parameters w and b are found by minimizing MSE. The tool for this is gradient descent, an iterative algorithm: start with an initial guess for w and b, compute the gradient (the direction that increases the loss fastest), and step in the opposite direction to reduce the loss. The size of each step is determined by a learning rate hyperparameter.

The lecture explains that choosing a good learning rate is essential. If it is too large, the updates can overshoot the minimum, causing the loss to bounce or even explode. If it is too small, progress is painfully slow and training takes a long time. Helpful strategies include starting small and increasing until improvement slows, then backing off, using learning rate schedules that gradually reduce the rate during training, or using adaptive methods like Adam that adjust the effective learning rate for each parameter automatically.

Linear regression is praised for being simple, interpretable, and computationally efficient. It is often recommended as a baseline: try it first, and if it performs sufficiently well, you may not need a more complex model. However, it has limitations. The most important one is that it assumes a linear relationship between features and the target, which is not always realistic. The lecture gives examples of non-linear relationships: electricity demand versus temperature tends to be high at both hot and cold extremes, which looks like a U-shaped curve rather than a straight line; tree height versus age shows rapid growth early and slower growth later, another clearly non-linear pattern. Linear regression is also sensitive to outliers due to the squared-error loss, and it can overfit when it memorizes training noise rather than learning general patterns.

To address overfitting, regularization adds a penalty to the loss that discourages overly complex models. The lecture presents two main types: L1 regularization (lasso) adds the sum of absolute values of the weights to the loss, promoting sparsity (many weights become exactly zero) and thus performing feature selection. L2 regularization (ridge) adds the sum of squared weights, shrinking weights toward zero but rarely turning them off completely, which stabilizes the fit and reduces variance. A hyperparameter lambda controls the strength of this penalty: higher lambda means a stronger push toward simpler models, while lower lambda allows more flexibility.

The lecture also introduces polynomial regression as an extension that can capture non-linear relationships without abandoning linear regression’s simplicity. The trick is to expand the input features by adding polynomial terms (e.g., x^2, x^3), and then apply standard linear regression on the expanded feature set. This keeps the model linear in the parameters but allows curved fits. Ridge regression (linear regression with L2 regularization) and lasso regression (linear regression with L1 regularization) are highlighted as practical tools to control complexity and improve generalization.

By the end, you understand the full loop of building a basic regression model: define the model form (linear), choose a loss (MSE), minimize it with gradient descent (tuning the learning rate), be aware of outliers and non-linearity, and use regularization to prevent overfitting. With these ideas in place, you are set up to move on to logistic regression for classification problems, where many of the same optimization and hyperparameter principles apply. The lecture’s structure flows from the problem setup (supervised regression), to the model and loss, to optimization via gradient descent, to practical concerns (learning rate, pros/cons, outliers), and finally to extensions (regularization and polynomial features), giving a complete and clear foundation for predictive modeling with linear methods.