Stanford CME295 Transformers & LLMs | Autumn 2025 | Lecture 7 - Agentic LLMs

This lecture teaches L1 regularization (also called LASSO), a powerful technique to prevent overfitting by adding a penalty to a model’s loss function that depends on the sum of absolute values of the model’s weights. Overfitting happens when a model fits training data extremely well but performs poorly on new, unseen data. The key idea behind regularization is to balance two goals at once: fit the data well and keep the model simple. L1 regularization creates simplicity by pushing many weights to be exactly zero, which automatically removes unhelpful features and makes the model easier to interpret.

The lecture starts by reminding you why we regularize at all: models with too many degrees of freedom can latch onto noise in the dataset. To fight this, we add a penalty to the loss, giving the objective function: objective = loss + λ × penalty. The parameter λ (lambda) controls how strong the penalty is; higher λ means stronger regularization, which usually shrinks or zeroes more coefficients. The lecture then contrasts two main types of penalties: L2 (Ridge), which uses the sum of squared weights, and L1 (LASSO), which uses the sum of absolute values. This difference might look small at first, but it leads to very different outcomes when optimizing.

A central part of the lecture is the geometric intuition. In two dimensions with weights w1 and w2, you can imagine the optimization as moving contour lines (like elevation lines on a map) of the loss function until they just touch a “constraint region.” For L2, the constraint region is a circle (because w1^2 + w2^2 ≤ c). For L1, the constraint region is a diamond (because |w1| + |w2| ≤ c). Circles are smooth, while diamonds have sharp corners that sit right on the axes. When the loss contours press against the diamond, they are more likely to touch at a corner, which makes one weight exactly zero. This geometric picture explains why L1 creates sparse solutions (lots of zeros), while L2 rarely does.

The lecture then discusses when to use each method. If you have many features and suspect many are irrelevant (common in high-dimensional problems), L1 is very useful because it performs feature selection automatically. If you have a moderate number of features and believe most matter at least a little, L2 is often better, as it shrinks all weights toward zero without turning them off completely. Sometimes you might want both effects: some sparsity and some stability. That is where Elastic Net comes in—a combined penalty that blends L1 and L2, controlled by a mixing hyperparameter.

Next, the talk covers practical pros and cons. Advantages of L1 include built-in feature selection, which leads to simpler, more interpretable models with fewer non-zero coefficients. This can also reduce storage and computation for prediction. Disadvantages include computational considerations: the absolute value function in L1 is not differentiable at zero (it has a sharp corner), so basic gradient descent doesn’t apply directly. Instead, you use methods like subgradient descent or coordinate descent, which can be slower than the smooth-gradient methods commonly used for L2. Another drawback is stability: L1 solutions can change a lot if the data changes slightly, especially when features are highly correlated, because different features may take turns being selected.

The instructor also answers common questions. Why is L1 computationally heavier? Because its penalty is non-differentiable at zero, forcing us to use slower methods like subgradient or coordinate descent. Why not just remove irrelevant features before training? You often don’t know which ones are irrelevant, or they may have tiny effects that are still useful; L1 can discover this automatically and keep small but real signals if λ isn’t overly large. Finally, choosing λ is critical and is typically done by cross-validation: you try different λ values, check validation performance, and pick the best balance.

By the end, you should be able to: explain why we regularize, write the L1 objective function, describe how λ controls the trade-off between fit and simplicity, explain geometrically why L1 sets coefficients to zero while L2 does not, decide when to use L1 vs L2 vs Elastic Net, and understand the computational differences and algorithms used for L1. The lecture is designed for beginners to intermediate learners who know basic linear models and loss functions. Prior knowledge helpful here includes linear regression, the idea of fitting a loss function, and basic optimization concepts like gradients. With these ideas, you can apply L1 to build simpler, more interpretable, and better-generalizing models.