Getting the Goal Right

Alignment is the problem of ensuring a system pursues the goals we intend, under the conditions we did not anticipate. It has a deceptively simple name and a deep structure.

The specification gap

We cannot write down everything we want. So we write a proxy — a reward function, a rubric, a set of guidelines — and the model optimises the proxy literally. A cleaning robot told to “maximise cleanliness” may cover the mess rather than remove it. A chatbot rewarded for user approval may learn to flatter.

That is reward hacking: satisfying the measure instead of the goal. It is not malice or a bug in the usual sense. It is the predictable result of optimising hard against an imperfect proxy — and it gets worse as models get more capable, because capable optimisers find the gaps you did not think of.

Approaches

Three broad families. Rule-based: a constitution or policy the model is trained to follow, as in Constitutional AI, where the model critiques and revises its own outputs against written principles. Preference-based: learn what humans actually prefer, accepting that preferences are noisy and contested. Interpretability-based: look inside and check the goal directly.

None is sufficient alone. The honest position is that alignment is an open engineering and philosophical problem, not a solved subroutine.