Tokenization

Why does Kannada text cost more tokens than English?

Write the same four-sentence message in English, Hindi and Kannada and pass each version through cl100k_base, the tokenizer of GPT-4. The English version becomes 32 tokens, the Hindi version 130 and the Kannada version 283. All three say the same thing, yet an API that bills per token charges almost nine times as much for the Kannada one, and it fills almost nine times as much of the model’s context. Code and JSON with long keys also cost more than plain English prose.

To see what a token is, start with a simpler question: how do you multiply the word “cat”?

You can’t, because computers cannot multiply words.

Everything you have built so far eats numbers: dot products, matrix multiplies, neurons. Text is not numbers. So before a language model can do anything at all, something has to turn your prompt into numbers.

It happens in three small hops:

Pause and predict The Hindi and Kannada versions say the same thing as the English one in many more tokens. What decides how many tokens a message costs?

The problem: what should one integer stand for?

Someone has to choose what the pieces are. There are two obvious choices: one integer per word, or one integer per character.

Pause and predict Which of the two would you pick for a language model?

The problem

Text must become a sequence of integers from a fixed list.

The naive fix

Use whole words (short sequences) or single characters (tiny list).

Why it fails

Words: the list is unbounded and unseen words have no ID. Characters: sequences are long and each piece means little.

The idea

Subwords. Frequent strings get their own ID. Rare strings are spelled out from smaller pieces that do have IDs.

The trade-off

Token boundaries follow frequency, not meaning or spelling. The model loses direct sight of letters and digits.

With subwords, a common word is one piece and a rare word is a few familiar pieces:

the   tokenization   unbelievableness

(An illustration. The exact cuts depend on the tokenizer, as you are about to see.)

These pieces have a name:

Token, vocabulary, token ID

A token is one piece of text from a fixed list. The list is the vocabulary. A token’s position in the list is its token ID, and that integer is all the model receives.

For example: Vocabulary: 0 = “a”, 1 = “t”, 2 = “at”, 3 = “c”. The text “cat” becomes the IDs [3, 2].

Nobody picks the strings that get their own ID by hand. An algorithm finds them by counting which strings occur often.

An algorithm you could have invented

Suppose you start with single characters and want longer pieces, built by gluing two pieces together at a time.

Pause and predict Which two characters would you glue together first?

1. Start the vocabulary with every single character in your training text.
2. Count every pair of neighbouring tokens. Find the most frequent pair.
3. Glue that pair into a new token. Add it to the vocabulary. Write the rule down.
4. Repeat until the vocabulary is as big as you want.

The only thing “learned” is the ordered list of glue rules, the merge list. To tokenize new text, split it into characters and replay the rules in the same order. To turn IDs back into text, concatenate the strings they stand for.

Notice that there is no neural network here, no gradient, no loss. Tokenizer training is a counting loop you could write in an afternoon. It runs once, before model training starts, and its result is a plain data file.

Let's see the numbers

Train BPE by hand on a short sentence:

That is 22 characters (spaces count), using 10 different ones. Count neighbouring pairs. “a” followed by “t” occurs 3 times (cat, sat, mat). Nothing else occurs more than twice. So merge 1 is a + t.

After merge 5 no pair occurs twice, so training stops: 10 characters + 5 merges = a vocabulary of 15. The same text went from 22 tokens to 11.

At merge 2 several pairs were tied at 2. The code takes the one it met first in the text, which was “t”, “h”. Any fixed rule works, as long as it is always the same rule.

Now use it on words it has never seen. Split into characters, then replay merges 1 to 5 in order:

• “chat” → c h a t → merge 1 applies → chat. Three tokens.
• “the hat” → merges 1, 2, 3, 4 apply → the hat. Three tokens for seven characters.

Type the sentence into the playground’s training box (untick “repeat”) and you will get exactly these five merges.

What you should remember

1. A model sees integers, not text. Text → tokens → token IDs happens before the neural network, and IDs → text after it.
2. Words fail (unbounded list, unseen words have no ID). Characters fail (long sequences, little meaning per piece). Subwords are the compromise.
3. BPE: repeatedly merge the most frequent adjacent pair. The trained tokenizer is the ordered merge list. To encode, replay the merges in the order they were learned. To decode, join the strings the IDs stand for.
4. Frequent strings become one token. Rare strings (typos, unusual names, under-represented languages) shatter into many tokens. Tokens are the unit of cost and context limits, and the reason models struggle with spelling, letter counting and long numbers.

Where this appears in a real LLM

In this lesson

• Trained on about 1,000 characters
• Starts from the characters it happened to see, so unseen characters have no ID
• A few dozen merges
• Merges can run across spaces, so whole phrases become tokens

In a real system

• Trained on many gigabytes of text
• Starts from the 256 byte values, so nothing is ever unknown
• Vocabularies of about 50,000 tokens (GPT-2), 128,000 (Llama 3) and 200,000 (OpenAI’s o200k_base, used from GPT-4o on)
• Text is first split by a pattern (roughly: words, numbers, punctuation), and merges never cross those splits

What this means for you as a developer

• Tokens are the unit of money and memory. APIs bill per token, and the context window is counted in tokens too: 128,000 tokens is a common size, and many frontier models now accept 1 million. English prose is about 4 characters per token. Code, JSON with long keys, and non-English text usually cost more. Larger vocabularies narrow the gap: on the four-sentence message from the top, o200k_base needs 32 tokens for English, 41 for Hindi and 58 for Kannada, against 130 and 283 with GPT-4’s cl100k_base.
• Spelling and letter counting are hard because the letters are hidden inside IDs. The same goes for reversing a string or rhyming by spelling.
• Long numbers are split into chunks whose boundaries have nothing to do with place value (for example “1234567” might arrive as 1234567). Column-wise arithmetic is awkward when you cannot see the columns. This is one reason models are handed calculators and code interpreters.
• Leading spaces matter. In GPT-style vocabularies “ the” (with a space) and “the” are different tokens with different IDs. A prompt that ends in a trailing space, or a stop-sequence that differs by one space, can behave surprisingly.

The token counts at the top now have a plain explanation. Bills count tokens, tokens follow the tokenizer’s training text, and that text was mostly English, so a Kannada message with the same meaning is cut into smaller pieces. Embeddings gives each token a meaning.