Stanford CS336 Language Modeling from Scratch | Spring 2025 | Lecture 1: Overview and Tokenization

This first lecture launches a hands-on course about building language models—the computer systems that assign probabilities to sequences of words and can predict the next word. The focus is on learning both the ideas and the practical skills to construct models from scratch. The class covers classic N-gram models, modern neural models (RNNs, LSTMs, Transformers), training processes, evaluation metrics, and especially the crucial preprocessing step: tokenization. Along the way, you will learn how to prepare data, select hyperparameters, and measure how well your model works.

The lecture starts with course goals: deeply understand language modeling; implement systems in Python and PyTorch; and master training and evaluation. You’ll see where language models are used every day—speech recognition, machine translation, text generation, and text classification. For example, in speech recognition, a language model helps choose between ambiguous sound interpretations like “recognize speech” versus “wreck a nice beach.” In translation, it pushes the system toward fluent, grammatical outputs. In generation, it produces new text that resembles the training data; and in classification, it helps distinguish categories like spam vs. not spam.

You learn what a language model does in precise terms: it estimates the probability of a word sequence, such as P(w1, w2, …, wn), and uses that to predict likely continuations, like the word after “the cat sat on the.” Classic N-gram models compute these probabilities from counts of short windows of words (like bigrams and trigrams). Neural language models go further by learning richer patterns and capturing long-distance relationships: RNNs process sequences step by step, LSTMs handle long-term dependencies, and Transformers use attention to focus on the most relevant words, defining today’s state-of-the-art.

Evaluation uses metrics that quantify how well a model performs. Perplexity measures how many choices a model effectively juggles on average at each step—the lower, the better. BLEU (often used for machine translation) compares model output to human references to gauge similarity and fluency. These metrics guide you to improve models and compare different approaches.

Before you can train any model, you must tokenize text—that is, break it into tokens. Naive whitespace splitting fails on real text because it mishandles punctuation and contractions. A better tokenizer splits “I’m going to the store.” into [“I”, “’m”, “going”, “to”, “the”, “store”, “.”]. Tokenization tools like NLTK and spaCy can handle these rules, and there are specialized tokenizers for domains like code and social media. You also learn about tokenization challenges: contractions, punctuation, and numbers (which come in many formats, like 1,000.00 and “one thousand”). Normalizing numbers can simplify the model’s job.

Subword tokenization addresses rare words and large vocabularies by breaking words into smaller pieces. For example, “unbelievable” becomes “un,” “believe,” and “able.” Common subword methods include Byte Pair Encoding (BPE), WordPiece, and SentencePiece. BPE starts with characters and merges the most frequent adjacent pairs repeatedly until reaching a desired vocabulary size. A toy example with words like “low,” “lower,” “newest,” and “widest” shows how frequent pairs such as “e”+“s” merge into “es,” and then “es”+“t” into “est.”

The lecture also covers course structure and logistics. You’ll have lectures, homework, and a final project; grading is 60% homework, 30% project, and 10% participation. Prerequisites include introductory programming (CS106A or equivalent) and probability/statistics (CS109 or equivalent); deep learning familiarity is helpful but not required. Resources include a course website with notes and assignments, Piazza for Q&A, recorded lectures, and office hours offered by the instructor and TAs (Alice, Bob, and Carol). You’ll code in Python and PyTorch and have access to GPUs for training. For extra learning, the PyTorch website and CS230 materials provide excellent tutorials.

Finally, you get a preview of what’s next: a deeper dive into N-gram language models, including smoothing, backoff, and interpolation—methods that handle the problem of unseen word sequences by sensibly redistributing probability. This foundation prepares you for the neural approaches covered later, culminating in modern transformer-based large language models.