Give it a fragment, get the whole thing back. That one move — run on
words instead of ink — is what a chatbot does to hold a conversation.
Its name is attention.
the two changes that get there, precisely
The move is the same; two things change on the way, and they are
not the same change.
A sharper energy. Equation [7] sums over
pairs of pixels; that quadratic is what caps it near
0.15 N. Score each stored page as a whole instead, and
capacity turns exponential — still in black and white:
E = −∑s exp(ξs⊤ x)
ξs is still stored page s; x is the page on
screen — both written as ±1 per pixel. Each term scores how well the screen matches one stored
page, and the exp lets the best match tower over the rest.
Then shades of grey, which buys differentiability —
not capacity — so the whole thing can sit inside a trained network.
Recall is now one line, with X the five stored pages set
side by side as columns:
xnew = X softmax(β X⊤ x)
The softmax turns the five match scores into weights that sum to
one — a gentler "pick the best" — and the new page is the stored
pages blended by those weights. β sets how sharply the best match dominates.
Feed the fragment and the memories through three learned
projections — query, key, value — set
β = 1/√dk, and
that same line is attention:
Z = softmax(QK⊤/√dk) V
The fragment is the query; the stored pages are the keys and values.
In a language model there is no fixed library — each word attends to
the other words in the same sentence, so the memories are
whatever it is reading.
The valley is still there — this energy has a floor too — but it lands
in a single step, where the 1982 rule you just ran
needed thousands of flips to reach it.