AI Foundationspredict · compress · act

concepts → Objective

Objective

Cross-entropy

timeless4 connections

the cost of coding data with the wrong model — the loss nearly everything trains on

the formal statement
H(p, q) (bits)
H(p, q) = −Σ p(x) log q(x)

Where it sits

The prism has six jobs across and eight layers down. Its primary cell islearn × L4, with a noted second cell. Hatched cells cannot exist — a GPU does not learn, an institution does not infer.

measure×L0 — it is itself an information-theoretic quantity

What must come first — and what it unlocks

Left to right is reading order, derived from the prerequisite_of edges. Nothing here is hand-ordered: the diagram is the graph.

EntropyCross-entropynothing depends on it yet — a leaf in the reading order

Before it: Entropy

Where to read it

The chapter that introduces it, and any chapter that uses it again.

07Distance Between Beliefsact 1 · The Substrate

Where it comes from

paperA Mathematical Theory of CommunicationClaude E. Shannon · 1948

Every connection

All 4 edges touching this node, grouped by relation family — the sections above are highlights from this list. Colours match the relation families inthe atlas.

Order · 1
requiresEntropyQuantity
Flow · 3
usesEntropyQuantity
is used byPerplexityQuantity
is used byPretrainingAlgorithm

This page is a projection of one node in src/data/concepts.ts. It has no prose file of its own — 230 declared edges produce all 236 of these pages. Edit an edge and both endpoints change.