Stanford CS230 | Autumn 2025 | Lecture 2: Supervised, Self-Supervised, & Weakly Supervised Learning

This lecture teaches decision trees, one of the most practical and understandable models in machine learning. A decision tree predicts an outcome by asking a sequence of simple questions about input features, like walking through a flowchart. You start at the root node, follow branches based on yes/no answers or threshold checks, and finish at a leaf node that outputs the prediction. The big idea is to choose questions that best separate the data into groups that are as pure as possible. Purity means that the examples in a group mostly share the same label, which makes the final decision more certain.

The lecture explains how trees choose the best questions using a concept called information gain, which is based on entropy. Entropy measures disorder: a mixed group of labels has high entropy, while a group where nearly all labels agree has low entropy. Information gain is the reduction in entropy after splitting on a feature. The model compares all possible splits and chooses the one that most reduces entropy, leading to more homogeneous child groups. A clear classroom example shows that splitting students by 'submitting assignments' better separates pass vs. fail than splitting by 'attending lectures.'

Two kinds of trees are covered: classification trees for categorical outcomes (like yes/no or types) and regression trees for continuous outcomes (like prices). The learning process, called recursive partitioning, keeps splitting the data into smaller subsets that are more uniform. This continues until stopping criteria are met, such as a maximum depth or a minimum number of samples in a leaf. These rules help control the complexity of the tree and reduce overfitting.

The lecture highlights strong advantages of trees: they are highly interpretable, handle both numerical and categorical data, and naturally capture non-linear relationships without feature scaling. However, it also addresses notable weaknesses: trees can overfit, can become complex and deep, and can be sensitive to small changes in data, producing different structures each time. To address these, pruning and constraints are recommended. Pruning removes weak branches that do not improve accuracy, and constraints limit depth or leaf size to keep the model simpler and more generalizable.

Beyond single trees, the lecture introduces ensemble methods that build on them. Random forests train many trees on different random samples and features, then average their predictions, making the final model more stable and accurate. Gradient boosting builds trees one after another, with each new tree focusing on fixing the errors the earlier trees made. Both methods often outperform a single tree when accuracy and robustness are priorities.

The target audience includes beginners and practitioners who want a clear and practical model they can explain to teammates or stakeholders. You should be comfortable with basic machine learning ideas like features and labels, but you do not need to know advanced math. After this lecture, you will be able to describe the structure of a decision tree, explain entropy and information gain in simple terms, understand the training process, and apply methods to prevent overfitting. You will also know when to choose a decision tree and when to use ensemble variants such as random forests or gradient boosting.

The lecture is structured from basic definitions and parts of a tree, to how trees choose splits, to learning via recursive partitioning, to advantages and disadvantages, and finally to practical fixes and ensemble improvements. Real-world examples (vision and glasses, student performance, ad clicks, house prices, healthcare, loans, and marketing) ground the theory in familiar contexts. The focus throughout is on clarity, interpretability, and practical decision-making about when and how to use tree-based models.