The Shortest Explanation
Chapter 2 made a large claim almost in passing: a model that compresses well has understood something, and one that merely stores has not. Chapter 12 invoked Occam’s razor and pointed out that it has to be paid for somewhere. This chapter is where those two meet, because there is a version of “prefer the simpler explanation” that is not a preference at all but a theorem about prediction.
Occam’s razor, made countable
Informally, Occam’s razor says the simpler explanation is more likely to be right. The trouble is that “simpler” is doing all the work and meaning nothing. Minimum description length (MDL), due to Jorma Rissanen, fixes that by measuring simplicity in bits.
The idea is a two-part bill. To describe a set of data using a hypothesis, you pay twice: once for the hypothesis, and once for the data given the hypothesis.
total cost = L(hypothesis) + L(data | hypothesis)A lookup table has a tiny hypothesis and a huge data term. A rule has a bigger hypothesis and a tiny data term. The best explanation is the one whose total is smallest — and that is a real optimisation, not a preference. Grünwald’s tutorial frames the whole principle as: the model that lets you write the data most briefly is the model to keep.
The ideal version, and why it is unreachable
Take this as far as it goes. Suppose the language you write descriptions in is a programming language, and you ask for the shortest program that outputs your data. That length is the Kolmogorov complexity of the data, and the strategy of preferring hypotheses with small Kolmogorov complexity is Solomonoff induction — arguably the best possible framework for prediction from data.
It is also impossible to carry out. Kolmogorov complexity is uncomputable: finding the shortest program that produces a given output is equivalent to the halting problem, and there is no algorithm that does it for all inputs. A general-purpose description language is too rich to search.
That is not a fatal objection, it is the reason practical MDL exists. You fix a specific description language — say, a neural network architecture with real-valued weights — and then “shortest description” becomes an ordinary cost you can minimise. The principle survives; the uncomputable ideal becomes an approximation.
Every training run is already doing this
Here is the connection that makes this chapter worth its place.
When a language model is trained on cross-entropy loss, the number it is minimising is the average number of bits needed to encode the next token. That is not a metaphor. The loss is a description length, and minimising it is minimising the code length of the corpus. Every gradient step in the guide so far has been an MDL calculation, carried out by a particular and very convenient description language: a transformer.
Which finally explains why chapter 2’s claim was not just a slogan. Compressing well and understanding are the same thing because predicting is coding — the two-part code is the bridge. A model that memorises the data pays the full data term; a model that finds the rule pays a small one.
What it buys, and what it does not
It buys a reason. Preferring simple models stops being superstition and becomes a consequence of coding: the simpler hypothesis is the cheaper predictor. Cheaper is what generalises.
It does not buy a method. MDL tells you which hypothesis is better once both are written down. It does not tell you how to find the short one — as the uncomputability of Kolmogorov complexity makes clear, that is the hard part and it is the part learning algorithms actually do.
MDL says which hypothesis to keep. It says nothing about what a network should throw away while it is learning — which parts of its input are irrelevant, and how much it should forget. That turns out to be the same problem viewed from the inside, and it has its own theory.