1Geometry (least squares)

Draw the line, or the surface, that comes closest to a cloud of points. Minimize the sum of squared deviations.

Lin ILin IICalc I
WHAT IS MADE SMALL
\[\hat{\beta} \;=\; \arg\min_{\beta} \; \lVert \mathbf{y} - X\beta \rVert^2\]

\(\mathbf{y}\) … the observed values, stacked into a vector
\(X\) … the explanatory variables arranged as a matrix (the design matrix)
\(\lVert\cdot\rVert^2\) … the sum of squares; the ruler that measures how large the deviation is

The solution is a system of linear equations

\[X^{\top}X\hat{\beta} = X^{\top}\mathbf{y}\]

The normal equations. They are solved by decomposing the matrix on the left (QR, Cholesky). This is the territory of Linear Algebra I.

The solution is an orthogonal projection

\[\hat{\mathbf{y}} = X(X^{\top}X)^{-1}X^{\top}\mathbf{y} = H\mathbf{y}\]

\(H\) is the projection matrix. The fitted values are the shadow cast by dropping the observations perpendicularly onto the space that the explanatory variables span.

Principal component analysis has the same idea behind it

\[\max_{\lVert w \rVert = 1} w^{\top}\Sigma w \quad\Longrightarrow\quad \Sigma w = \lambda w\]

Look for the direction in which the variance is largest, and you arrive at an eigenvector of the covariance matrix.

k-means also minimizes a sum of squares

\[\min_{C,\mu} \sum_{k}\sum_{i \in C_k} \lVert x_i - \mu_k \rVert^2\]

Make the total distance from the centre of each grouping small. No probability model is set up.

Sessions that cover this: Statistics I, session 10, Statistics I, session 13

Methods from actual research that fall under this perspective

In research: Gower's distance — building a distance between villages from nominal-scale accent types

In research: Correspondence analysis — placing rows and columns on one plane under the chi-square distance

In research: Phylogenetic networks (Neighbor-Net) — splitting a distance matrix, allowing the mixing to show

In research: Principal component analysis — seeking the direction of maximum variance leads to an eigenvalue problem

In research: Betweenness centrality — measuring how hard a village is to reach from shortest paths through the road network

In research: Speaker-normalized pitch-accent distance — comparing accent patterns while allowing time to stretch

2Frequentist (sampling variation, estimation, testing)

The parameter of the population is a fixed value; what is random is the sample. The quality of an estimator or a test is judged by how it varies when the sample is drawn again. Maximum likelihood is a much-used principle for choosing a parameter, but likelihood is used in Bayesian work as well.

Calc ICalc IILin I
WHAT IS MADE LARGE
\[\hat{\theta} \;=\; \arg\max_{\theta} \; \ell(\theta), \qquad \ell(\theta) = \sum_{i=1}^{n} \log p(x_i \mid \theta)\]

\(p(x_i\mid\theta)\) … the probability, or density, of that observation under the parameter \(\theta\)
\(\ell(\theta)\) … the log-likelihood; logarithms turn the product into a sum
\(\hat{\theta}\) … the maximum likelihood estimate, the value that makes the likelihood largest

The condition for a maximum is that the derivative is zero

\[\frac{\partial \ell(\theta)}{\partial \theta} = 0\]

The left-hand side is called the score function. Where it cannot be solved in closed form, it is solved numerically, by Newton's method among others.

How much an estimate varies is set by the information

\[I(\theta) = -\,\mathbb{E}\!\left[\frac{\partial^2 \ell}{\partial \theta^2}\right], \qquad \sqrt{n}\,(\hat{\theta}-\theta) \;\xrightarrow{d}\; N\!\left(0,\, I(\theta)^{-1}\right)\]

The larger the second derivative — the curvature — the steadier the estimate. Confidence intervals come out of here.

Assume normal errors and it coincides with the geometric view

\[p(y \mid X,\beta) \propto \exp\!\left(-\frac{\lVert \mathbf{y}-X\beta \rVert^2}{2\sigma^2}\right)\]

Take logarithms and it becomes exactly the minimization of a sum of squares. This is why least squares and maximum likelihood give the same answer.

What this perspective evaluates is the hit rate of a procedure

\[ \Pr\big(\theta \in \mathrm{CI}(X_1,\dots,X_n)\big) = 0.95 \qquad \text{(the proportion over repeated samples)} \]

The probability attaches to the interval; \(\theta\) does not move. The type I error rate of a test is defined the same way, as a proportion over repetitions of the procedure.

Sessions that cover this: Statistics I, session 7, Statistics II, session 7

Methods from actual research that fall under this perspective

In research: Beta regression — modelling a proportion without letting predictions leave the interval

In research: Rarefaction — levelling sample size before comparing lexical richness

In research: Power simulation — could the effect have been detected, had it been there

In research: Mantel test — testing the correspondence of two distance matrices by permuting rows and columns together

In research: Linear mixed models and AIC — nested observations by maximum likelihood, and choosing among models

In research: Multiple regression and variable selection — reading collinearity with VIF, choosing among all subsets

3Bayesian

Combine what was known before observing with the data, and update belief. Here it is the parameter that carries a distribution.

Calc ICalc IILin I
THE UPDATE
\[p(\theta \mid D) \;=\; \frac{p(D \mid \theta)\, p(\theta)}{p(D)}, \qquad p(D)=\int p(D\mid\theta)p(\theta)\,d\theta\]

\(p(\theta)\) … the prior; belief before the data are seen
\(p(D\mid\theta)\) … the likelihood, the same quantity the frequentist view was maximizing
\(p(\theta\mid D)\) … the posterior; belief after the update

Predictions also come out as distributions

\[p(\tilde{x} \mid D) = \int p(\tilde{x}\mid\theta)\, p(\theta \mid D)\, d\theta\]

Not a single value but a distribution over the data still to come. The prediction carries its uncertainty with it.

The integral in the denominator is hard, so it is approximated

\[\log p(D) \;\ge\; \mathbb{E}_{q}\!\left[\log p(D,\theta) - \log q(\theta)\right] \;=\; \mathrm{ELBO}(q)\]

Variational inference looks for the \(q\) that makes this lower bound largest. MCMC replaces the integral with samples.

With finitely many hypotheses, addition and division are enough

\[ P\!\left(p=\tfrac34 \,\middle|\, \text{H, H, H}\right) = \frac{(3/4)^3}{(1/4)^3+(3/4)^3} = \frac{27}{28} \]

Taking the probability of heads to be either \(1/4\) or \(3/4\), with a prior of \(1/2\) on each. The denominator is not an integral but a sum over finitely many terms. At this stage the update from prior through likelihood to posterior can be carried out before any calculus has been studied. Move to a continuous parameter and the integral returns, and with it the need to approximate.

Ridge regression is a Bayesian MAP estimate

\[\arg\max_{\beta} \left[\log p(\mathbf{y}\mid\beta) + \log p(\beta)\right] \;=\; \arg\min_{\beta}\left[\lVert \mathbf{y}-X\beta\rVert^2 + \lambda\lVert\beta\rVert^2\right]\]

Place a normal distribution as the prior and the expression becomes penalized least squares (\(\lambda = \sigma^2/\tau^2\)).

Sessions that cover this: Statistics I, session 12, Statistics II, session 3

Methods from actual research that fall under this perspective

In research: Bayesian estimation (Stan / brms) — priors, convergence diagnostics, and reading a posterior

In research: ICAR spatial random effects — putting adjacency into the prior

In research: Accentual class merger — modelling merger itself as mutation (method and implementation: Takuya Takahashi)

4Learning theory (risk minimization)

What is to be made small is the loss on data not yet seen, not the loss on the data at hand. No probability model need be set up.

Calc IILin ILin II
WHAT WE ACTUALLY WANT SMALL
\[R(f) \;=\; \mathbb{E}_{(x,y)\sim P}\left[\ell\big(f(x),\,y\big)\right] \qquad \text{(the expected risk)}\]

\(P\) … the distribution the data follow; its contents are unknown, but it is taken to exist
\(\ell\) … the loss function; the pain of a prediction that misses
\(f\) … the function being learned (the model)

Only the empirical risk can be computed from what is at hand

\[\hat{R}(f) = \frac{1}{n}\sum_{i=1}^{n} \ell\big(f(x_i),\,y_i\big), \qquad \hat{f} = \arg\min_{f \in \mathcal{F}} \hat{R}(f)\]

This is empirical risk minimization. Learning is taken to be this minimization and nothing more.

A generalization bound holds down the gap

\[R(f) \;\le\; \hat{R}(f) \;+\; \underbrace{\mathfrak{C}(\mathcal{F},\,n,\,\delta)}_{\text{model complexity term}}\]

The more freedom the model has, the larger the term on the right becomes. Regularization is the act of holding that term down.

The SVM makes the margin as wide as possible

\[\min_{w,b}\ \tfrac{1}{2}\lVert w\rVert^2 + C\sum_{i=1}^{n}\xi_i \quad \text{s.t.}\quad y_i(w^{\top}x_i+b) \ge 1-\xi_i,\ \ \xi_i \ge 0\]

Take the distance from the boundary to the nearest point as wide as it will go. A classifier that assumes nothing about probability.

Sessions that cover this: none at present

Methods from actual research that fall under this perspective

In research: Leave-one-out sensitivity — matching the unit left out to the unit of dependence

In research: Random forests — averaging weakly correlated trees to cut variance alone

5Information theory

Treat a short encoding as a good explanation. It gives a ruler for choosing between models.

Calc ICalc II
HOW MUCH UNCERTAINTY THERE IS
\[H(p) \;=\; -\sum_{x} p(x)\log p(x), \qquad D_{\mathrm{KL}}(p \,\|\, q) \;=\; \sum_{x} p(x)\log\frac{p(x)}{q(x)} \;\ge\; 0\]

\(H(p)\) … the entropy; how hard that distribution is to predict
\(D_{\mathrm{KL}}\) … the code length wasted when the true distribution \(p\) is described by the model \(q\)
equality … reached only when the two distributions agree

Cross-entropy is the same thing as maximum likelihood

\[H(p,q) = H(p) + D_{\mathrm{KL}}(p\,\|\,q) \qquad \Longleftrightarrow \qquad \min_{q} H(p,q) \;\equiv\; \max_{q}\ \text{likelihood}\]

Using cross-entropy as the loss of a classifier is doing maximum likelihood estimation.

AIC balances fit against complexity

\[\mathrm{AIC} = -2\,\ell(\hat{\theta}) + 2k\]

\(k\) is the number of parameters. A better fit counts in a model's favour, but the parameters it took are subtracted back out.

A decision tree splits on how much information is gained

\[\mathrm{IG} = H(\text{parent}) - \sum_{c}\frac{n_c}{n}H(\text{child}_c)\]

Measure how far the uncertainty fell by splitting, and take the split that lowers it most.

Sessions that cover this: Statistics I, session 11

Methods from actual research that fall under this perspective

In research: Normalized compression distance — measuring closeness by how much compresses away

6Games and adversaries

Read learning as a balance between two sides in competition. What helps one side hurts the other.

Calc IILin II
MINIMIZING AND MAXIMIZING AT ONCE
\[\min_{G}\ \max_{D}\ V(D,G) \;=\; \mathbb{E}_{x\sim p_{\text{data}}}\big[\log D(x)\big] \;+\; \mathbb{E}_{z\sim p_z}\big[\log\big(1-D(G(z))\big)\big]\]

\(G\) … the generating side; it wants to make fakes that pass for real
\(D\) … the discriminating side; it wants to tell real from fake
\(\min\max\) … each side answers the other's best move with its own best move; the point of balance is the solution

Preparing for the worst case has the same structure

\[\min_{\theta}\ \max_{Q \in \mathcal{U}}\ \mathbb{E}_{Q}\big[\ell(\theta)\big]\]

Make small the loss under the least convenient of the distributions one is willing to entertain (distributionally robust optimization).

Audio deepfakes have the same shape

\[\text{generator improves} \;\longrightarrow\; \text{detector weakens} \;\longrightarrow\; \text{detector improves}\]

Forgery and detection become a chase that does not end. Evaluating either side on its own means nothing.

Sessions that cover this: Speech Processing, session 13 (in preparation)

Methods from actual research that fall under this perspective

In research: Modelling cultural dynamics with a GAN — shifting the balance of adversarial loss and feature reward over time

In research: Economic games and cooperative behaviour — reading accent as a social tag to see whom people cooperate with

7Causality

Ask what would happen under an intervention. A different kind of question from prediction, and one the data alone do not settle.

Lin ILin IICalc I
HANDLING AN OUTCOME THAT CANNOT BE SEEN
\[\mathrm{ATE} \;=\; \mathbb{E}\big[Y(1) - Y(0)\big]\]

\(Y(1)\) … the outcome had the treatment been given
\(Y(0)\) … the outcome had it not been
the problem … for any one person, only one of the two can ever be observed

An assumption that is unusable without randomization

\[\big(Y(1),\,Y(0)\big) \;\perp\!\!\!\perp\; T \;\mid\; X\]

That the assignment of treatment is independent of the outcome once the covariates \(X\) are conditioned on. In observational data this is an assumption, and it cannot be checked.

The adjustment formula

\[\mathbb{E}\big[Y \mid do(T=t)\big] \;=\; \sum_{x} \mathbb{E}\big[Y \mid T=t,\, X=x\big]\, P(X=x)\]

Look at the effect within each stratum, then add them up weighted by the size of the stratum. Simpson's paradox is a problem of this weighting.

Adjustment by regression is a projection

\[\hat{\tau} = \big(Z^{\top}X\big)^{-1}Z^{\top}\mathbf{y} \quad \text{(instrumental variables, 2SLS)}\]

Another variable \(Z\) is brought in to avoid confounding. The computation comes down, in the end, to linear algebra.

Sessions that cover this: Statistics II, session 10, Statistics II, session 11

Methods from actual research that fall under this perspective

In research: The apparent-time method — reading generational difference as a proxy for change over time

The same equation reads differently from another perspective

Ridge regression　In the geometric view, penalized least squares. In the Bayesian view, a MAP estimate with a normal distribution placed as the prior. In learning theory, an operation that holds down the complexity of the model. All three point at the same expression, \(\min_\beta \lVert \mathbf{y}-X\beta\rVert^2 + \lambda\lVert\beta\rVert^2\).
Cross-entropy　It is the negative log-likelihood of maximum likelihood, it is the minimization of the KL divergence in information theory, and in learning theory it is one loss function among others.
The SVM　In learning theory, the maximization of a margin. Seen from the side of games, a min–max problem that prepares for the worst case.

The framework follows “Mathematical techniques and their relation to statistical methods and machine learning” (Onohara, September 2025), a set of notes Onohara wrote privately in order to think through the curriculum of the Faculty of Information Science. It speaks for no one but its author. The equations are given in representative form only; conditions and regularity assumptions are left out.
Which session falls under which perspective is in Methods & Mathematics; the ideas themselves are explained in A Guide to Statistical Thinking.