📚TheoryAdvanced

In-Context Learning Theory

Key Points

•
In-context learning (ICL) means a model learns from examples provided in the input itself, without updating its parameters.
•
Transformers can approximate classic learning algorithms like gradient descent and ridge regression purely within their forward pass.
•
Softmax attention often behaves like kernel regression, weighting labels by similarity between the query and context examples.
•
Ridge regression has a closed-form solution that transformers can implicitly approximate from sequences of input tokens.
•
Gradient descent can be simulated in-context by performing a few update-like computations on the prompt before predicting.
•
The theory of ICL connects attention to kernels and shows when transformers can implement learning rules on the fly.
•
Computation cost is dominated by attention (O( $n^{2}$ d)), but explicit algorithmic emulators like ridge involve matrix ops (up to O( $d^{3}$ )).
•
Good prompting (ordering, normalization, and consistent formats) greatly improves in-context algorithmic behavior.

Prerequisites

→Linear Algebra — Matrix multiplication, transposes, and inverses underpin ridge regression and attention computations.
→Optimization Basics — Understanding gradient descent and loss functions is key for algorithmic ICL via iterative updates.
→Probability and Statistics — Noise models and generalization intuition motivate regularization and averaging in kernel methods.
→Transformer Architecture — Self-attention, K/Q/V projections, and positional encodings are the computational substrate of ICL.
→Kernel Methods — Attention can act like a learned kernel; familiarity clarifies the ICL-kernel connection.

Detailed Explanation

Tap terms for definitions

01Overview

Hook: Imagine handing a calculator a few example math problems and, without changing its software, it figures out the pattern and solves a new problem immediately. Concept: That is the essence of in-context learning (ICL): a model learns from examples in its input, not by updating its weights. For transformers, the forward pass can internally compute something resembling a learning algorithm on the provided context. Example: Given a few (x, y) pairs from a line y = ax + b, a transformer can infer a and b from the prompt and then predict y for a new x. ICL theory studies when and how transformers can implement learning rules—like gradient descent steps or ridge regression—during a single forward pass. Rather than storing a universal mapping from inputs to outputs, the network reuses its fixed parameters to read the examples, compute statistics (e.g., correlations), and apply them to the query. This view explains why longer, well-structured prompts can noticeably improve performance and why attention mechanisms resemble classical nonparametric estimators. Researchers have shown equivalences between attention and kernel methods, and between stacked attention/MLP layers and approximate solvers for linear regression or iterative optimizers. In practice, transformers trained on diverse next-token prediction objectives acquire an inductive bias that encourages such on-the-fly computation. Understanding this helps you design better prompts and anticipate when ICL will generalize to new tasks.

02Intuition & Analogies

Hook: Think of a chef who can cook a new dish after tasting just a few samples, without rewriting their recipes. They adapt on the spot using experience. Concept: In-context learning is that chef-like adaptation for models: the model sees examples in the prompt and figures out the rule without changing its internal weights. Analogy 1: Flashcards. You show several input–output pairs. The model notices patterns—like matching colors or simple formulas—and then answers a new card using the same rule. Analogy 2: A toolbox. The transformer has fixed tools (attention heads, MLPs). When you give it examples, it chooses the right tools and assembles a quick plan (an internal algorithm) to solve your query. Why attention helps: Attention measures similarity. If your query looks a lot like one example, the model should predict a similar label—just like nearest-neighbor or kernel regression. With many heads and layers, the model can combine similarities, compute averages, fit a small linear model, or even approximate a couple of gradient steps. Ordering and formatting matter because they guide how the model groups and aggregates information—like neat notes versus a messy notebook. Example: Provide three pairs from y = 2x + 1: (1,3), (2,5), (3,7), then ask for y when x = 10. A good ICL-enabled transformer effectively computes line-fitting statistics in the forward pass (such as averages and covariances) and outputs 21, even though no parameter update happened. This perspective demystifies why more examples, consistent delimiters, and clean structure often yield better predictions: they make the internal, on-the-fly computation easier.

03Formal Definition

Hook: To formalize, we treat the prompt as a small training set and the query as a test point. Concept: A model f_

θ

implements in-context learning if its output on a query x_* depends on the empirical risk minimization (or a related learning rule) computed over the in-prompt context C = {(

x_{i}

y_{i}

)} without changing parameters

θ

. Definition: Let

A

be a learning algorithm (e.g., ridge regression), mapping a context C to a predictor

h_{C}

. A transformer f_

θ

exhibits ICL for

A

if there exists a mapping g_

θ

such that f_

θ

(C, x_*)

\approx

h_{C}

(x_*), where

h_{C}

A

(C). In words, the forward pass approximates the computation of

A

on C and then applies it to x_*. For ridge regression,

h_{C}

(x) = x^

⊤

(X^

⊤

X +

λ

I)^{-1} X^

⊤

y. For gradient descent,

h_{C}

is the predictor after T steps:

w^{(t+1)} = w^{(t

)} -

η

\nabla

L

(

w^{(t)}

; C), and

h_{C}

(x) = x^

⊤

w^{(T)}

. Attention can approximate kernel regression:

\hat

y(x_*) =

\sum_{i}

α_{i}

(x_*)

y_{i}

with

α_{i}

\propto

exp

(q^

⊤

k_{i}

d

). A sequence model trained for next-token prediction can implement g_

θ

using attention (to compute similarities and aggregates) and MLPs (to transform and combine statistics). The ICL property is distribution-dependent: f_

θ

may approximate

A

well on families of tasks present during pretraining (e.g., linear relationships) but not necessarily on out-of-distribution tasks.

04When to Use

Hook: Use ICL when you want rapid adaptation without retraining. Concept: ICL shines when small task descriptions fit naturally into prompts and latency constraints make parameter updates impractical.

Few-shot prediction: Provide k labeled examples and a query to get a task-specific prediction (classification or regression) on the fly.
Structured tasks that resemble simple algorithms: Linear relations, lookups, arithmetic patterns, or small program-like rules the model can infer from examples.
Rapid prototyping and personalization: Include a mini-profile or glossary in the prompt so the model tailors outputs instantly.
Multi-task settings: The prompt can declare the task via examples, allowing the same model to switch behavior without fine-tuning.
Low-resource or privacy-sensitive scenarios: Keep data local in prompts rather than storing it permanently in model weights. However, if you must handle large datasets, strict reliability guarantees, or continuous updates across many queries, traditional training or fine-tuning may be better. ICL is most effective when prompts are concise, well-formatted, and represent the task distribution the model has seen (or can easily approximate) during pretraining.

⚠️Common Mistakes

Hook: Many failures come from treating ICL like magic rather than an algorithm running inside the forward pass. Concept: Treat prompts as data for a temporary learner; then avoid pitfalls as you would in normal ML.

Messy prompt structure: Inconsistent delimiters, mixed formats, or shuffled fields make it hard for attention to align examples. Use clear separators and consistent ordering.
Expecting out-of-distribution generalization: If your task departs far from what the model has implicitly learned (e.g., highly nonlinear relations unseen in pretraining), ICL may fail. Validate with held-out prompts.
Too few or too many examples: Too few can underfit; too many can overrun context length or dilute relevant signals. Start small and scale until performance plateaus.
Ignoring normalization: Feature scales matter. Without normalization, attention and internal solvers can behave poorly. Normalize inputs in the prompt (e.g., z-score or min–max) and explain the convention.
Overinterpreting correctness: ICL is approximate. The model may mimic the right algorithm only partially (e.g., one or two GD steps). Use checks, sanity tests, and confidence estimates.
Assuming parameter updates: ICL does not change weights. If persistent learning is needed, use fine-tuning or retrieval-augmented designs.
Not leveraging order effects: Group similar examples together; place the most relevant examples closest to the query to exploit recency and attention biases.

Key Formulas

Softmax Attention Weights

α_{i} (q) = \frac{exp ( \frac{q ^{⊤} k _{i}}{d} )}{\sum _{j = 1}^{n} exp ( \frac{q ^{⊤} k _{j}}{d} )}

Explanation: Each weight measures similarity between the query q and key $k_{i}$ , normalized to sum to 1. Higher dot products produce larger weights, letting the model focus on relevant examples.

Nadaraya–Watson Estimator

\overset{y}{^} (x_{*}) = i = 1 \sum n α_{i} (x_{*}) y_{i}

Explanation: The prediction at x_* is a weighted average of labels, where weights come from a similarity kernel. In transformers, softmax attention often plays the role of these weights.

Ridge Regression Solution

\overset{w}{^}_{ridge} = (X^{⊤} X + λ I)^{- 1} X^{⊤} y

Explanation: This closed-form solution fits linear predictors with L2 regularization. It stabilizes learning when features are correlated or data are scarce.

Gradient Descent Update

w^{(t + 1)} = w^{(t)} - η \nabla L (w^{(t)}; C)

Explanation: Parameters move opposite the gradient of the loss over the context C, scaled by learning rate $η$ . A few steps can be emulated by a forward pass to achieve in-context adaptation.

Mean Squared Error

L_{MSE} (w) = \frac{1}{n} i = 1 \sum n (x_{i}^{⊤} w - y_{i})^{2}

Explanation: The average squared difference between predictions and labels. Minimizing MSE leads to the normal equations and connects to ridge when regularized.

Kernel Ridge Regression Prediction

\overset{y}{^} (x_{*}) = k_{*}^{⊤} (K + λ I)^{- 1} y, k_{*} = [k (x_{*}, x_{i})]_{i = 1}^{n}

Explanation: Predictions are computed using the kernel vector between the query and training points and the inverse of the regularized kernel matrix. Attention can approximate this with learned kernels.

Self-Attention Cost

T_{attn} (n, d) = O (n^{2} d), S_{attn} (n) = O (n^{2})

Explanation: For sequence length n and hidden size d, attention computes all pairwise token interactions. This quadratic behavior often dominates the cost of ICL.

Bias via Augmented Features

\tilde{X} = [1, X], \overset{w}{^} = (\tilde{X}^{⊤} \tilde{X})^{- 1} \tilde{X}^{⊤} y

Explanation: Adding a column of ones lets linear models learn an intercept. It is a standard trick used in both closed-form and gradient-based solvers.

Linear Model with Gaussian Noise

y = Xw + ε, ε \sim N (0, σ^{2} I)

Explanation: This probabilistic model underlies least squares as a maximum likelihood estimator, motivating ridge as a MAP estimator with a Gaussian prior on w.

Complexity Analysis

When transformers perform in-context learning, the dominant computational cost arises from self-attention. For a sequence length n (number of tokens, which grows with the number of examples and formatting tokens) and hidden size d, computing all dot products

q_{i}

k_{j}

and the weighted sums costs O(

n^{2}

d). Memory for attention logits and probabilities is O(

n^{2}

). Feed-forward layers add O(n

d^{2}

) time and O(n d) memory. Thus, longer prompts quadratically increase compute and memory. If the forward pass internally approximates a classic algorithm, the implied complexity can differ. For example, explicit ridge regression with d features requires building

X^{T}

X in O(n

d^{2}

) time and inverting a d × d matrix in O(

d^{3}

). In a learned, implicit implementation, the network uses fixed parameters to approximate similar computations across layers; the end-to-end complexity still follows the transformer’s O(

n^{2}

d) budget rather than exposing an O(

d^{3}

) inversion directly, though some heads/layers may effectively encode matrix–vector operations. When simulating gradient descent in-context, running T synthetic steps conceptually adds O(T nd) work; however, within a transformer, these steps are “unrolled” into layers and attention heads. Empirically, small T (1–3) is often sufficient for simple tasks, keeping effective cost moderate. For attention-as-kernel regression (Nadaraya–Watson), the main expense is similarity computation with all context points, again matching O(

n^{2}

d). In summary: the theoretical algorithm the model emulates (ridge, GD, kernel methods) informs what patterns it can learn and how many examples it needs, but the runtime and memory footprint are governed primarily by the transformer architecture—quadratic in prompt length and linear in hidden size per attention head.

Code Examples

Ridge Regression Inside a Forward Pass (Closed-Form ICL Emulator)

1 #include <bits/stdc++.h>
2 using namespace std;
3 
4 // A tiny matrix helper for small dimensions (educational, not optimized)
5 struct Mat {
6     int r, c;
7     vector<double> a; // row-major
8     Mat(int r=0, int c=0, double v=0.0): r(r), c(c), a(r*c, v) {}
9     double& operator()(int i, int j){ return a[i*c + j]; }
10     double  operator()(int i, int j) const { return a[i*c + j]; }
11 };
12 
13 Mat transpose(const Mat& M){
14     Mat T(M.c, M.r);
15     for(int i=0;i<M.r;++i) for(int j=0;j<M.c;++j) T(j,i) = M(i,j);
16     return T;
17 }
18 
19 Mat multiply(const Mat& A, const Mat& B){
20     assert(A.c == B.r);
21     Mat C(A.r, B.c, 0.0);
22     for(int i=0;i<A.r;++i)
23         for(int k=0;k<A.c;++k){
24             double aik = A(i,k);
25             for(int j=0;j<B.c;++j)
26                 C(i,j) += aik * B(k,j);
27         }
28     return C;
29 }
30 
31 Mat identity(int n){ Mat I(n,n,0.0); for(int i=0;i<n;++i) I(i,i)=1.0; return I; }
32 
33 Mat add(const Mat& A, const Mat& B){
34     assert(A.r==B.r && A.c==B.c);
35     Mat C(A.r,A.c);
36     for(int i=0;i<A.r*A.c;++i) C.a[i] = A.a[i] + B.a[i];
37     return C;
38 }
39 
40 Mat scale(const Mat& A, double s){ Mat B=A; for(double &x:B.a) x*=s; return B; }
41 
42 // Gaussian elimination for matrix inverse (for small d)
43 Mat inverse(Mat A){
44     assert(A.r == A.c);
45     int n = A.r;
46     Mat I = identity(n);
47     // Augment A | I
48     Mat aug(n, 2*n, 0.0);
49     for(int i=0;i<n;++i){
50         for(int j=0;j<n;++j) aug(i,j) = A(i,j);
51         for(int j=0;j<n;++j) aug(i,n+j) = I(i,j);
52     }
53     // Row-reduce
54     for(int col=0; col<n; ++col){
55         // Pivot
56         int pivot = col;
57         for(int i=col+1;i<n;++i) if (fabs(aug(i,col)) > fabs(aug(pivot,col))) pivot = i;
58         if (fabs(aug(pivot,col)) < 1e-12) throw runtime_error("Singular matrix");
59         if (pivot != col) for(int j=0;j<2*n;++j) swap(aug(col,j), aug(pivot,j));
60         // Normalize row
61         double diag = aug(col,col);
62         for(int j=0;j<2*n;++j) aug(col,j) /= diag;
63         // Eliminate others
64         for(int i=0;i<n;++i){ if(i==col) continue; double factor = aug(i,col);
65             for(int j=0;j<2*n;++j) aug(i,j) -= factor * aug(col,j);
66         }
67     }
68     // Extract inverse
69     Mat inv(n,n);
70     for(int i=0;i<n;++i) for(int j=0;j<n;++j) inv(i,j) = aug(i,n+j);
71     return inv;
72 }
73 
74 // Compute ridge weights: w = (X^T X + lambda I)^{-1} X^T y
75 vector<double> ridge_weights(const Mat& X, const vector<double>& y, double lambda){
76     int n = X.r, d = X.c;
77     Mat Xt = transpose(X);
78     Mat XtX = multiply(Xt, X);
79     Mat reg = identity(d);
80     reg = scale(reg, lambda);
81     Mat A = add(XtX, reg);
82     Mat Ainv = inverse(A);
83     // X^T y as d x 1
84     Mat ymat(n,1);
85     for(int i=0;i<n;++i) ymat(i,0) = y[i];
86     Mat Xty = multiply(Xt, ymat);
87     Mat wmat = multiply(Ainv, Xty);
88     vector<double> w(d);
89     for(int j=0;j<d;++j) w[j] = wmat(j,0);
90     return w;
91 }
92 
93 // Predict y for a single x using weights w
94 double predict_linear(const vector<double>& x, const vector<double>& w){
95     assert((int)x.size() == (int)w.size());
96     double s=0.0; for(size_t i=0;i<x.size();++i) s += x[i]*w[i]; return s;
97 }
98 
99 int main(){
100     // Context: y ≈ 2x + 1 with small noise; we use augmented features [1, x]
101     vector<pair<double,double>> ctx = {{1,3.1},{2,5.0},{3,7.0},{4,9.2}};
102     int n = (int)ctx.size();
103     int d = 2; // [bias, x]
104     Mat X(n,d);
105     vector<double> y(n);
106     for(int i=0;i<n;++i){
107         X(i,0) = 1.0;         // bias
108         X(i,1) = ctx[i].first; // x
109         y[i]   = ctx[i].second; // y
110     }
111     double lambda = 1e-3; // small regularization
112     vector<double> w = ridge_weights(X, y, lambda);
113 
114     // Query: predict y when x = 10
115     vector<double> xq = {1.0, 10.0};
116     double yhat = predict_linear(xq, w);
117 
118     cout.setf(std::ios::fixed); cout<<setprecision(4);
119     cout << "Learned weights (bias, slope): " << w[0] << ", " << w[1] << "\n";
120     cout << "Prediction for x=10: " << yhat << "\n";
121     return 0;
122 }
123

This program emulates an ICL forward pass that performs ridge regression on context pairs and then predicts the query. We build X with a bias term, compute (X^T X + λI)^{-1} X^T y via small matrix utilities, and evaluate x*. In theory, a transformer can approximate this computation internally with attention and MLP layers.

Time: O(nd^2 + d^3) for building X^T X and inverting the d×d matrixSpace: O(d^2 + nd)

Simulating In-Context Gradient Descent Steps Before Prediction

1 #include <bits/stdc++.h>
2 using namespace std;
3 
4 // Perform T steps of GD on MSE over context to fit linear w, then predict query
5 struct GDICL {
6     double eta; int T; // learning rate and steps
7     GDICL(double eta=0.05, int T=50): eta(eta), T(T) {}
8 
9     // x: n x d, y: n, returns predicted y for xq after T steps
10     double fit_and_predict(const vector<vector<double>>& x, const vector<double>& y, const vector<double>& xq){
11         int n = (int)x.size();
12         int d = (int)x[0].size();
13         vector<double> w(d, 0.0); // start from zero (acts like a prior)
14         auto dot = [](const vector<double>& a, const vector<double>& b){
15             double s=0; for(size_t i=0;i<a.size();++i) s+=a[i]*b[i]; return s; };
16         for(int t=0;t<T;++t){
17             // Compute gradient of MSE: (2/n) * X^T (Xw - y)
18             vector<double> grad(d, 0.0);
19             for(int i=0;i<n;++i){
20                 double err = dot(w, x[i]) - y[i];
21                 for(int j=0;j<d;++j) grad[j] += err * x[i][j];
22             }
23             for(int j=0;j<d;++j) grad[j] = (2.0/n) * grad[j];
24             for(int j=0;j<d;++j) w[j] -= eta * grad[j];
25         }
26         return dot(w, xq);
27     }
28 };
29 
30 int main(){
31     // Context: y = -3x + 5 (augmented features [1, x])
32     vector<vector<double>> X = {{1,0},{1,1},{1,2},{1,3}};
33     vector<double> y = {5.0, 2.0, -1.0, -4.0};
34     vector<double> xq = {1.0, 4.0};
35     GDICL solver(0.1, 200);
36     double yhat = solver.fit_and_predict(X, y, xq);
37     cout.setf(std::ios::fixed); cout<<setprecision(4);
38     cout << "Prediction for x=4 after in-context GD: " << yhat << "\n";
39     return 0;
40 }
41

This example simulates a transformer that performs a few gradient steps on the prompt examples before answering. We iteratively minimize MSE to obtain a linear predictor and evaluate it on the query. In ICL theory, a few steps of GD can be encoded by layers that transform and aggregate context information.

Time: O(T n d) for T gradient steps over n examples of dimension dSpace: O(nd + d)

Attention as Kernel Regression (Nadaraya–Watson Estimator)

1 #include <bits/stdc++.h>
2 using namespace std;
3 
4 // Stable softmax
5 vector<double> softmax(const vector<double>& z){
6     double m = *max_element(z.begin(), z.end());
7     vector<double> e(z.size()); double s=0.0;
8     for(size_t i=0;i<z.size();++i){ e[i] = exp(z[i]-m); s+=e[i]; }
9     for(size_t i=0;i<z.size();++i) e[i] /= max(s, 1e-18);
10     return e;
11 }
12 
13 double attention_kernel_predict(const vector<vector<double>>& X, const vector<double>& y,
14                                 const vector<double>& xq, double beta){
15     // logits = beta * (xq^T xi)
16     auto dot = [](const vector<double>& a, const vector<double>& b){
17         double s=0; for(size_t i=0;i<a.size();++i) s+=a[i]*b[i]; return s; };
18     int n = (int)X.size();
19     vector<double> logits(n);
20     for(int i=0;i<n;++i) logits[i] = beta * dot(xq, X[i]);
21     vector<double> w = softmax(logits); // attention weights
22     double pred = 0.0; for(int i=0;i<n;++i) pred += w[i]*y[i];
23     return pred;
24 }
25 
26 int main(){
27     // 1D regression in augmented space [1, x] so dot-products include a bias-like effect
28     vector<vector<double>> X = {{1, -2},{1,-1},{1,0},{1,1},{1,2}};
29     vector<double> y = {4,1,0,1,4}; // roughly y ≈ x^2 with small bias; kernel smoother will average nearby labels
30     vector<double> xq = {1, 1.5};
31     double d = (double)X[0].size();
32     double beta = 1.0 / sqrt(d); // temperature similar to transformers
33     double yhat = attention_kernel_predict(X, y, xq, beta);
34     cout.setf(std::ios::fixed); cout<<setprecision(4);
35     cout << "Attention-kernel prediction at x=1.5: " << yhat << "\n";
36     return 0;
37 }
38

This code implements a single-head attention-like predictor: compute dot-product similarities between the query and context, softmax them, and return a weighted average of labels. It mirrors the Nadaraya–Watson estimator and illustrates how attention can realize kernel regression inside a forward pass.

Time: O(n d) to compute logits and O(n) to normalizeSpace: O(n) for weights

1	#include <bits/stdc++.h>
2	using namespace std;
3
4	// A tiny matrix helper for small dimensions (educational, not optimized)
5	struct Mat {
6	int r, c;
7	vector<double> a; // row-major
8	Mat(int r=0, int c=0, double v=0.0): r(r), c(c), a(r*c, v) {}
9	double& operator()(int i, int j){ return a[i*c + j]; }
10	double operator()(int i, int j) const { return a[i*c + j]; }
11	};
12
13	Mat transpose(const Mat& M){
14	Mat T(M.c, M.r);
15	for(int i=0;i<M.r;++i) for(int j=0;j<M.c;++j) T(j,i) = M(i,j);
16	return T;
17	}
18
19	Mat multiply(const Mat& A, const Mat& B){
20	assert(A.c == B.r);
21	Mat C(A.r, B.c, 0.0);
22	for(int i=0;i<A.r;++i)
23	for(int k=0;k<A.c;++k){
24	double aik = A(i,k);
25	for(int j=0;j<B.c;++j)
26	C(i,j) += aik * B(k,j);
27	}
28	return C;
29	}
30
31	Mat identity(int n){ Mat I(n,n,0.0); for(int i=0;i<n;++i) I(i,i)=1.0; return I; }
32
33	Mat add(const Mat& A, const Mat& B){
34	assert(A.r==B.r && A.c==B.c);
35	Mat C(A.r,A.c);
36	for(int i=0;i<A.r*A.c;++i) C.a[i] = A.a[i] + B.a[i];
37	return C;
38	}
39
40	Mat scale(const Mat& A, double s){ Mat B=A; for(double &x:B.a) x*=s; return B; }
41
42	// Gaussian elimination for matrix inverse (for small d)
43	Mat inverse(Mat A){
44	assert(A.r == A.c);
45	int n = A.r;
46	Mat I = identity(n);
47	// Augment A \| I
48	Mat aug(n, 2*n, 0.0);
49	for(int i=0;i<n;++i){
50	for(int j=0;j<n;++j) aug(i,j) = A(i,j);
51	for(int j=0;j<n;++j) aug(i,n+j) = I(i,j);
52	}
53	// Row-reduce
54	for(int col=0; col<n; ++col){
55	// Pivot
56	int pivot = col;
57	for(int i=col+1;i<n;++i) if (fabs(aug(i,col)) > fabs(aug(pivot,col))) pivot = i;
58	if (fabs(aug(pivot,col)) < 1e-12) throw runtime_error("Singular matrix");
59	if (pivot != col) for(int j=0;j<2*n;++j) swap(aug(col,j), aug(pivot,j));
60	// Normalize row
61	double diag = aug(col,col);
62	for(int j=0;j<2*n;++j) aug(col,j) /= diag;
63	// Eliminate others
64	for(int i=0;i<n;++i){ if(i==col) continue; double factor = aug(i,col);
65	for(int j=0;j<2n;++j) aug(i,j) -= factor aug(col,j);
66	}
67	}
68	// Extract inverse
69	Mat inv(n,n);
70	for(int i=0;i<n;++i) for(int j=0;j<n;++j) inv(i,j) = aug(i,n+j);
71	return inv;
72	}
73
74	// Compute ridge weights: w = (X^T X + lambda I)^{-1} X^T y
75	vector<double> ridge_weights(const Mat& X, const vector<double>& y, double lambda){
76	int n = X.r, d = X.c;
77	Mat Xt = transpose(X);
78	Mat XtX = multiply(Xt, X);
79	Mat reg = identity(d);
80	reg = scale(reg, lambda);
81	Mat A = add(XtX, reg);
82	Mat Ainv = inverse(A);
83	// X^T y as d x 1
84	Mat ymat(n,1);
85	for(int i=0;i<n;++i) ymat(i,0) = y[i];
86	Mat Xty = multiply(Xt, ymat);
87	Mat wmat = multiply(Ainv, Xty);
88	vector<double> w(d);
89	for(int j=0;j<d;++j) w[j] = wmat(j,0);
90	return w;
91	}
92
93	// Predict y for a single x using weights w
94	double predict_linear(const vector<double>& x, const vector<double>& w){
95	assert((int)x.size() == (int)w.size());
96	double s=0.0; for(size_t i=0;i<x.size();++i) s += x[i]*w[i]; return s;
97	}
98
99	int main(){
100	// Context: y ≈ 2x + 1 with small noise; we use augmented features [1, x]
101	vector<pair<double,double>> ctx = {{1,3.1},{2,5.0},{3,7.0},{4,9.2}};
102	int n = (int)ctx.size();
103	int d = 2; // [bias, x]
104	Mat X(n,d);
105	vector<double> y(n);
106	for(int i=0;i<n;++i){
107	X(i,0) = 1.0; // bias
108	X(i,1) = ctx[i].first; // x
109	y[i] = ctx[i].second; // y
110	}
111	double lambda = 1e-3; // small regularization
112	vector<double> w = ridge_weights(X, y, lambda);
113
114	// Query: predict y when x = 10
115	vector<double> xq = {1.0, 10.0};
116	double yhat = predict_linear(xq, w);
117
118	cout.setf(std::ios::fixed); cout<<setprecision(4);
119	cout << "Learned weights (bias, slope): " << w[0] << ", " << w[1] << "\n";
120	cout << "Prediction for x=10: " << yhat << "\n";
121	return 0;
122	}
123

1	#include <bits/stdc++.h>
2	using namespace std;
3
4	// Perform T steps of GD on MSE over context to fit linear w, then predict query
5	struct GDICL {
6	double eta; int T; // learning rate and steps
7	GDICL(double eta=0.05, int T=50): eta(eta), T(T) {}
8
9	// x: n x d, y: n, returns predicted y for xq after T steps
10	double fit_and_predict(const vector<vector<double>>& x, const vector<double>& y, const vector<double>& xq){
11	int n = (int)x.size();
12	int d = (int)x[0].size();
13	vector<double> w(d, 0.0); // start from zero (acts like a prior)
14	auto dot = [](const vector<double>& a, const vector<double>& b){
15	double s=0; for(size_t i=0;i<a.size();++i) s+=a[i]*b[i]; return s; };
16	for(int t=0;t<T;++t){
17	// Compute gradient of MSE: (2/n) * X^T (Xw - y)
18	vector<double> grad(d, 0.0);
19	for(int i=0;i<n;++i){
20	double err = dot(w, x[i]) - y[i];
21	for(int j=0;j<d;++j) grad[j] += err * x[i][j];
22	}
23	for(int j=0;j<d;++j) grad[j] = (2.0/n) * grad[j];
24	for(int j=0;j<d;++j) w[j] -= eta * grad[j];
25	}
26	return dot(w, xq);
27	}
28	};
29
30	int main(){
31	// Context: y = -3x + 5 (augmented features [1, x])
32	vector<vector<double>> X = {{1,0},{1,1},{1,2},{1,3}};
33	vector<double> y = {5.0, 2.0, -1.0, -4.0};
34	vector<double> xq = {1.0, 4.0};
35	GDICL solver(0.1, 200);
36	double yhat = solver.fit_and_predict(X, y, xq);
37	cout.setf(std::ios::fixed); cout<<setprecision(4);
38	cout << "Prediction for x=4 after in-context GD: " << yhat << "\n";
39	return 0;
40	}
41

1	#include <bits/stdc++.h>
2	using namespace std;
3
4	// Stable softmax
5	vector<double> softmax(const vector<double>& z){
6	double m = *max_element(z.begin(), z.end());
7	vector<double> e(z.size()); double s=0.0;
8	for(size_t i=0;i<z.size();++i){ e[i] = exp(z[i]-m); s+=e[i]; }
9	for(size_t i=0;i<z.size();++i) e[i] /= max(s, 1e-18);
10	return e;
11	}
12
13	double attention_kernel_predict(const vector<vector<double>>& X, const vector<double>& y,
14	const vector<double>& xq, double beta){
15	// logits = beta * (xq^T xi)
16	auto dot = [](const vector<double>& a, const vector<double>& b){
17	double s=0; for(size_t i=0;i<a.size();++i) s+=a[i]*b[i]; return s; };
18	int n = (int)X.size();
19	vector<double> logits(n);
20	for(int i=0;i<n;++i) logits[i] = beta * dot(xq, X[i]);
21	vector<double> w = softmax(logits); // attention weights
22	double pred = 0.0; for(int i=0;i<n;++i) pred += w[i]*y[i];
23	return pred;
24	}
25
26	int main(){
27	// 1D regression in augmented space [1, x] so dot-products include a bias-like effect
28	vector<vector<double>> X = {{1, -2},{1,-1},{1,0},{1,1},{1,2}};
29	vector<double> y = {4,1,0,1,4}; // roughly y ≈ x^2 with small bias; kernel smoother will average nearby labels
30	vector<double> xq = {1, 1.5};
31	double d = (double)X[0].size();
32	double beta = 1.0 / sqrt(d); // temperature similar to transformers
33	double yhat = attention_kernel_predict(X, y, xq, beta);
34	cout.setf(std::ios::fixed); cout<<setprecision(4);
35	cout << "Attention-kernel prediction at x=1.5: " << yhat << "\n";
36	return 0;
37	}
38