Vectors and the dot product

How do you measure whether two lists of numbers agree?

A retrieval system holds 200,000 documents, and each one is stored as a list of 768 numbers. A question arrives and is turned into a list of 768 numbers too. Before the user notices any delay, the system has to give every one of the 200,000 documents a single score that says how well its list agrees with the question’s list.

Shrink the lists to three numbers to see how that score is computed. Say the question is q = [−1, +1, 0] and one document is d = [+1, −1, 0].

Pause and predict Multiply the two lists slot by slot and add the three products. What is the total?

The same comparison runs inside the language model itself, and it is the operation the model runs more than any other. When you type “What is a cat?”, the model first replaces each token, “cat” included, with a list of a few thousand numbers it learned during training. Every step after that compares lists with other lists, millions of times for every word the model writes, and each comparison asks the same question: do these two lists of numbers agree?

Attention asks “which earlier words matter for this word?”. Similarity search asks “which stored document matches this question?”. Next-word scores ask “how well does each possible next word fit here?”. Each answer is an agreement score between two lists.

All three are the same small operation, the dot product, and you will meet it in almost every lesson that follows.

The problem

In The surprising idea you saw that an LLM is a function that turns numbers into numbers.

So any “thinking” it does about similarity has to be arithmetic. Your first instinct as a programmer is an equality check, and that does not get far.

The problem

Given two lists of numbers, produce a single number that says how much they agree.

The naive fix

Compare them like a programmer would: a == b, or count how many slots are equal.

Why it fails

Two learned lists are never exactly equal. We need a graded answer (a lot, a little, not at all, the opposite), and it has to be cheap enough to run millions of times per word.

The idea

Multiply the lists slot by slot and add everything up. Matching signs push the total up, clashing signs pull it down.

The trade-off

A list with big numbers gets a big score just for being big. We fix that at the end of this lesson, with cosine similarity.

A mental model

First, the thing itself. You have used it for years under another name.

Vector

A list of numbers. In code you would call it an array, a row, or a record.

For example: person = [34, 95000, 172] is age, salary, height. A “3-dimensional vector” just means the list has 3 numbers in it.

A list of two numbers has a bonus: you can draw it. [3, 1] means “3 steps right, 1 step up”.

Draw an arrow from the origin to that point. Now the list has a direction and a length.

Two arrows can point the same way, at right angles to each other, or in opposite directions. The dot product is a single number that tells you which of these is the case, and by how much.

Dot product

A way of asking two vectors: how much do your directions agree? One number comes out. Large positive: they agree. Zero: they have nothing to say about each other. Negative: they oppose.

For example: Two arrows pointing the same way give a big positive number. Two arrows at right angles give exactly 0. Two arrows pointing in opposite directions give a negative number.

Let's see the numbers

Here is the playground’s starting position by hand. A = [3, 1] and B = [2, 4].

Pause and predict What is A · B?

Nothing changes with more numbers

Here are vectors with three slots, taken from the course repository. Think of them as made-up “meaning” numbers for three words:

You cannot draw a 3-number arrow on this page, let alone a 768-number one, and you do not need to. The recipe and its meaning stay the same: agree, unrelated, oppose.

Why divide out the lengths?

Imagine a search. Your question is q = [2, 1]. Two documents compete:

By raw dot product, B wins, although A is a perfect match in direction and B won only by being long.

To stop that, divide the dot product by both lengths. What is left is pure direction: cosine similarity.

The math

You have done all of this already. Here it is in the notation papers use.

What you should remember

1. A vector is a list of numbers. With two numbers you can draw it as an arrow: it has a direction and a length.
2. The dot product asks “how much do these two directions agree?”. Arrows pointing the same way give a large positive number, arrows at a right angle give 0, and opposite arrows give a negative number.
3. The recipe is multiply matching slots, add up. It is identical in 2, 4 or 4,096 dimensions. Only the drawing stops working.
4. Cosine similarity is the dot product with both lengths divided out, always from −1 to +1, so a long vector cannot win just by being long.
5. Attention scores, similarity search and next-word scores are all dot products. You will meet this operation in almost every remaining lesson.

Where this appears in a real LLM

In this lesson

• 2 numbers per vector, so we can draw arrows
• Numbers picked by hand
• One dot product at a time

In a real system

• Hundreds to thousands of numbers per vector
• Every number learned from data; single slots rarely have a nameable meaning
• Millions of dot products per generated token (billions of individual multiplications), grouped into matrix multiplies (next lesson)

Keep the dot product and the cosine apart. Attention and the output layer use the raw dot product, so there a vector’s length does count, and the model is free to use it.

Dividing the lengths out is a choice. Similarity search often makes it. Attention only divides by a fixed number (the square root of the vector size), though some recent models also normalise the query and key vectors first.