AI Foundationspredict · compress · act

concepts → Mechanism

Mechanism

Self-supervised learning

timeless2 connectionsdraft

hide part of the data and let the model teach itself to recover it

Where it sits

The prism has six jobs across and eight layers down. Its primary cell islearn × L4. Hatched cells cannot exist — a GPU does not learn, an institution does not infer.

What must come first — and what it unlocks

Left to right is reading order, derived from the prerequisite_of edges. Nothing here is hand-ordered: the diagram is the graph.

nothing comes first — this is a starting pointSelf-supervised learningPretrainingLanguage model

Unlocks: Pretraining · Language model

Where to read it

The chapter that introduces it, and any chapter that uses it again.

26The Big Readact 6 · The Attention Revolution

Where it comes from

paperBERT: Pre-training of Deep Bidirectional Transformers for Language UnderstandingJacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova · 2018

Every connection

All 2 edges touching this node, grouped by relation family — the sections above are highlights from this list. Colours match the relation families inthe atlas.

Order · 2
must come beforePretrainingAlgorithm
must come beforeLanguage modelArchitecture

This page is a projection of one node in src/data/concepts.ts. It has no prose file of its own — 230 declared edges produce all 236 of these pages. Edit an edge and both endpoints change.