Embeddings

Why does “cat” sit next to “car”?

The tokenizer from Tokenization turns “cat” into token 464, “dog” into token 12 and “car” into token 465. By these numbers, “cat” sits right next to “car”, and far away from “dog”.

A metro map works the other way. On Bengaluru’s Purple Line, MG Road, Trinity and Halasuru are neighbours on the map because they are neighbours in the city. The map means something. The token IDs mean nothing at all.

The IDs were handed out in the order the tokenizer happened to create them. An ID is like a primary key in a database. It tells you which row you have. It tells you nothing about what the row is like.

And “what is this token like?” is exactly what a model needs. A model that cannot tell that cats and dogs are similar must learn every fact about them twice.

Pause and predict “cat” is token 464, “car” is 465 and “dog” is 12. What should the model use instead of the ID, so that cat and dog can look alike?

The problem: arithmetic on a name

Your first instinct as a programmer may be to feed the ID straight in. It is already a number, and networks take numbers.

Pause and predict What goes wrong if a network does arithmetic on raw token IDs?

The problem

Token IDs are arbitrary labels, but the network does arithmetic on its inputs.

The naive fix

Feed the ID in as a number, or describe each word by hand-written features (is_animal, is_plural, …).

Why it fails

Raw IDs invent a fake order and size. Hand-written features would need millions of human judgements, and nobody knows which features matter.

The idea

Give every token its own row of adjustable numbers in a table. Start them random. Let training move them.

The trade-off

One more big matrix of parameters to learn, and the numbers in it have no human-readable meaning.

A mental model: words as points on a map

Draw two dots close together and label them “cat” and “dog”. Put a third, “car”, in the far corner. Tokens that are used alike sit close together, and unrelated tokens sit far apart. That is what an embedding gives you.

You already describe things with lists of numbers. A house in a property database might be [price, area, bedrooms, year]. Two similar houses have similar lists. Nobody wrote a similar() function. Similarity falls out of the representation.

An embedding does this for tokens. Each token becomes a point in space. The space is arranged so that distance and direction mean something: “cat” near “dog”, both far from “car”.

Embedding

A list of numbers that stands for a token, chosen (by training) so that tokens used in similar ways get similar lists.

For example: cat → [−3.0, 3.2], dog → [−3.6, 2.4], car → [−2.4, −3.4]. cat and dog are 1.0 apart; cat and car are 6.6 apart.

Pause and predict Token 464 comes in. How does the model get its vector?

Let's see the numbers

First, the lookup. A tiny vocabulary of 4 tokens, 3 numbers each. E has 4 rows:

The embedding of token 2 is E[2] = [0.8, 0.9, 0.2]. Getting it is an ordinary array index: the model reads row 2 and does no arithmetic at all.

You will often read that this “is a matrix multiplication”. It is, in disguise. First write token 2 as a one-hot vector: a row of zeros with a single 1 in it, at position 2. (“One-hot” means one entry is switched on and all the rest are off.)

Pause and predict What do you get when you multiply the row [0, 0, 1, 0] by E?

Real code uses the index, because multiplying 50,000 numbers by zero is a waste. The matrix view matters for one reason: it shows that the table is an ordinary layer of weights, so backpropagation can train it like any other layer.

Second, similarity. With the toy coordinates from the explorer:

The question “are these two tokens alike?” has become a multiplication and a sum, which a matrix multiply can do for every pair at once.

The math

Two small formulas: the lookup, and the fake task that trains it.

And the skip-gram guessing game, built from parts you already own:

When training is over, W is thrown away and E is what you keep. (The original word2vec avoided this full softmax over the vocabulary, which is expensive, with a cheaper approximation called negative sampling. The idea is the same.)

What you should remember

1. A token ID is a name, not a quantity, so an embedding layer is a lookup table: E[token_id], one learned row per token. It equals one-hot × matrix, so it trains like any other layer.
2. Nobody assigns the coordinates. They are learned from a prediction task. Tokens used in similar contexts are pushed toward similar vectors.
3. Similarity is measured with the dot product / cosine. Real spaces have hundreds to thousands of dimensions because words are alike in many independent ways. The 2D picture is only a teaching model.
4. A lookup gives one static vector per token, whatever the sentence. The word “bank” shows the flaw: it needs a different vector in “river bank” and in “bank account”. A later lesson fixes this.

Where this appears in a real LLM

In a GPT the embedding table is simply the first layer of the network. In the repository’s tiny_gpt.py it is one line:

(pos_emb adds information about word order. That is a later lesson.)

In this lesson

• 70 words, 32 numbers each
• Trained separately, on a fake task (predict a neighbouring word)
• 28 sentences
• The output matrix W is thrown away

In a real system

• 30,000 to 260,000 tokens, 768 to 16,384 numbers each
• No separate step: the table is trained jointly with every other layer, on next-token prediction
• Trillions of tokens of text
• The same table is often reused at the top of the model to score the next token. That trick is called weight tying, and you will build it in Build a GPT

The mechanism is the same: rows of a matrix, looked up by token ID, shaped by gradients from a prediction task. Modern LLMs do not run word2vec. They do not need to, because next-token prediction also pushes similar tokens toward similar rows, though in an LLM most of the meaning in context is built by the later layers.

You will meet “embedding models” again when we build retrieval (RAG). Those produce one vector for a whole sentence or document, so that search becomes nearest-neighbour lookup. The idea is the same, applied to a bigger unit. Token embeddings (this lesson) are an internal layer of the LLM. Sentence embeddings are a separate model’s output.

The loose end: one vector per token is not enough

Where did you put “bank”? Any single spot is wrong. In “the river bank” it should sit with water. In “the money bank” it should sit with finance. A lookup table cannot see the sentence, so “bank” gets one vector: an awkward average that is wrong in every actual sentence.

What we need is a way for a token’s vector to be adjusted by the tokens around it. That mechanism is Attention, the centre of every Transformer. First, we need a model that actually predicts something.