Matrices: many dot products at once

Why is almost everything in an LLM a matrix multiply?

Open the source code of any LLM and search for the character @, Python’s matrix multiply operator. You will find lines like x @ W, Q @ K.T and weights @ V in every layer. GPT-2 small has 124 million parameters, and writing one token costs roughly one multiplication and one addition per parameter, so about 124 million of each. Almost all of that arithmetic happens on the lines with an @.

If you deleted those lines, almost nothing would be left. To a surprising degree, an LLM is a long chain of one operation, the matrix multiply.

Pause and predict The dot product takes two lists of numbers and returns one number. What do you guess a matrix multiply is?

One dot product asks one question of one vector. A model needs to ask thousands of questions of thousands of tokens, at every layer. A matrix multiply is how you write, and run, all of those at once.

The problem

Suppose a layer wants to compute 64 different scores for each of 1,000 token vectors. Every score is a dot product. That is 64,000 dot products.

The first version every programmer writes uses two loops: one over the token vectors and one over the scoring formulas.

The problem

Run many scoring formulas over many vectors.

The naive fix

Two nested for loops, calling dot() 64,000 times.

Why it fails

In Python that is painfully slow. Worse, it hides something important: none of those 64,000 dot products depends on any other. They could all run at the same time, but a loop says “one after another”.

The idea

Stack the vectors into one grid, stack the formulas into another grid, and define a single operation that means “dot everything on the left with everything on the right”.

The trade-off

You have to keep track of the shapes of the grids. Shape mismatches are the most common bug in ML code. Fortunately they behave like type errors: the code refuses to run, and the shapes tell you why.

A mental model

Matrix

A grid of numbers. If a vector is one row of a spreadsheet, a matrix is the whole sheet. You can also see it as a stack of vectors.

For example: Three tokens, each described by 2 numbers: a grid with 3 rows and 2 columns. We say its shape is (3, 2). Always rows first.

You will read matrices in two ways, and switch between them all the time:

Matrix × vector: several dot products at once

A house is described by x = [2, 1, 1]: bedrooms, bathrooms, garages. We have two scoring formulas, one per row:

So W @ x = [4, 4]: two questions asked of the same vector give two answers.

W has shape (2, 3), x has 3 numbers, the output has 2. A layer with 64 outputs is a matrix with 64 rows: 64 questions asked at once.

Keep these two formulas in mind. You will meet them again below, written the other way round.

Matrix × matrix: every row with every column

Now put several vectors on the right-hand side, standing up as columns. The rule:

In A @ B, the number in row i, column j of the result is (row i of A) · (column j of B). Every row of A gets dotted with every column of B, and the definition needs nothing else.

Let's see the numbers

Pause and predict A has shape (2, 2) and B has shape (2, 3). What shape is A @ B?

Six dot products, arranged in a grid, make up the full result.

The shape rule

Row i of A and column j of B must have the same length, or the dot product cannot be formed. Rows of A are as long as A has columns. Columns of B are as tall as B has rows. So:

A tiny “layer”: three tokens at once

In LLM code the data usually stands on the left: one row per token. Here are 3 tokens with 2 numbers each, pushed through a weight matrix that turns 2 numbers into 3.

Check token 2: [1, 1] · [2, 1] = 3, [1, 1] · [0, 3] = 3, [1, 1] · [1, 0] = 1.

Each row of the result depends only on the matching row of X. The tokens do not mix. The same transformation is applied to each of them, independently, in one go.

Notice the switch. In the house example the questions were the rows of W, and we wrote W @ x. Here the data is on the left, so each question is a column of W, and we write X @ W. The dot products are the same; they are only laid out sideways. LLM code almost always uses this second form. The next section shows, with the house numbers, that the two are the same thing.

The math

The transpose: flip the spreadsheet

One more tool, and it is purely about layout. The transpose turns rows into columns. It is written A^T in papers and A.T in code. A (2, 3) matrix becomes (3, 2). No number changes, only where it sits.

Why would you want that? Take the three token vectors X from above and ask: how similar is every token to every other token? Similarity is a dot product, so we want all 9 pairs.

Try the obvious thing first. X @ X is (3, 2) @ (3, 2), and the inner numbers are 2 and 3. They do not match, so it cannot run.

Here is why. A matrix multiply dots rows of the left with columns of the right, but our tokens are sitting in rows on both sides. So flip the right-hand copy and stand its tokens up as columns:

something @ something_else.T is how you spell “dot every row of this with every row of that”. Attention is built on this pattern, where it appears as Q @ K.T.

The bridge: W @ x and x @ W are the same layer

Now the transpose pays for the switch you saw earlier. Take the house again: x = [2, 1, 1] and the two formulas as rows of W.

Now lay x down as a row, the way LLM code stores tokens, and flip W so each formula stands up as a column:

The numbers and the answers are the same; only the layout changed. Stack a second house [3, 2, 0] under the first, and one multiply scores both:

So a layer stored as (inputs, outputs), with one column per question, is the transpose of the “rows are questions” matrix. Code stores it that way from the start and writes x @ W. When the Neurons lesson says “each column of W is one neuron”, this is why.

What you should remember

1. A matrix is a grid of numbers: a stack of vectors. Its shape is (rows, columns), rows first, always.
2. Matrix × vector = one dot product per row. Each row asks its own question of the vector.
3. Matrix × matrix: cell (i, j) = row i of A · column j of B. It is many dot products computed at once.
4. Shape rule: (n, k) @ (k, m) → (n, m). Inner numbers must match, outer numbers are the answer. Treat mismatches like type errors.
5. The transpose flips rows and columns. X @ Y.T means “every row of X dotted with every row of Y”.

Where this appears in a real LLM

In this lesson

• 2 × 3 grids of whole numbers
• 6 dot products per multiply
• Numbers typed in by you

In a real system

• In GPT-2 small, one feed-forward weight matrix has shape (768, 3072): 2,359,296 learned numbers
• A prompt of T tokens goes through it as one (T, 768) @ (768, 3072) multiply
• 12 layers in GPT-2 small and dozens in larger models, several such matrices per layer, for every token generated

You will see these exact patterns again: x @ W in every layer, Q @ K.T in attention, and a final multiply that produces one score per word in the vocabulary, which is where the next lesson picks up.