Models that see and hear

How does the bot read a ticket that is only a photo?

A support ticket arrives with no text at all.

It is a photo of a phone screen showing a red circle with a white cross and the words “Payment failed.” Below the photo the customer has typed one line: “then why is money gone from my account??”

Support tickets often look like this: screenshots, photos of bank SMS messages, blurry pictures of receipts. Customers tend to show the problem instead of typing it.

A common guess is that a model which takes images runs OCR behind the scenes, pulls out the text, and reads that. Look at what a language model accepts. The one you built in Build GPT eats one thing only: a list of token IDs, each turned into a row of an embedding table. There is no row for “red circle”.

Pause and predict Your model eats token IDs, each turned into a row of an embedding table, and there is no row for “red circle”. What must happen before it can read the photo?

The problem: a photo is not a sentence

Count the numbers in that screenshot. A phone screen is 1080 × 2400 pixels, and each pixel has three colour numbers (red, green and blue).

Pause and predict How many numbers is that one screenshot?

Text had a natural unit, the token, and a fixed vocabulary. A picture has neither. So what should one image token be?

The problem

A language model needs a short sequence of vectors, and an image is millions of numbers with no vocabulary.

The naive fix

Convert the image to text first, or feed every pixel as a token.

Why it fails

Text loses everything that is not words. Pixels make sequences far too long, and each one carries almost no meaning.

The idea

Cut the image into squares of, say, 14 × 14 pixels. Flatten each square into a list of numbers and multiply it by a learned matrix. Each square becomes one vector: one token. No vocabulary needed.

The trade-off

An image still costs hundreds or thousands of tokens, and very small details inside one square can be blurred together.

A mental model

Here is the whole idea as one picture. Follow the highlighted square.

Patch embedding

Cut the image into equal squares (patches). Write each square’s pixels out as one long list of numbers. Multiply that list by a learned matrix to get a vector as wide as the model’s tokens. Add a vector that says where the square was.

For example: A 14 × 14 patch has 14 × 14 × 3 = 588 numbers. A 588 × D matrix turns them into D numbers: one image token.

Notice what is missing: there is no lookup. A text token ID picks a row from a table. A patch is its own row: its pixels go straight into a matrix multiply. That is why an image never has an “unknown token”.

Three parts: encoder, projector, language model

Most open vision-language models (for example the LLaVA family) are built from three pieces:

The vision encoder is itself a Transformer, the same kind you built, but trained on images. Its job is to turn raw patch numbers into vectors that describe what is in each part of the picture, in context.

How the encoder learns what things look like

Where does such an encoder come from? The best-known recipe is CLIP (OpenAI, 2021). It used 400 million pairs of an image and its caption from the web.

Two encoders are trained side by side: one for images, one for text. For a batch of pairs, the goal fits in one sentence. The vector of each image should be close to the vector of its own caption, and far from the other captions in the batch.

Contrastive training

Learn by comparison: pull matching pairs together, push mismatched pairs apart. No one labels “this is a cat”. The caption is the label.

For example: A photo of a cat and the caption “a cat asleep on a sofa” should get a high dot product. The same photo and “a bus in the rain” should get a low one.

After this training, an image encoder produces vectors that line up with language: a photo of a failed-payment screen lands near text about errors and payments. CLIP and SigLIP encoders are the “eyes” of many open vision-language models.

Plugging the eyes into the language model

The encoder’s vectors have the wrong width and live in a different “space” from the language model’s embeddings. The projector fixes that. In the original LLaVA (2023) it was a single matrix. LLaVA-1.5 used a small two-layer MLP.

Training usually goes in two stages. First, freeze the encoder and the language model and train only the projector on image-caption pairs, so image tokens start to “look like” words to the language model. Then fine-tune on conversations about images (questions, answers, descriptions).

The other design: natively multimodal models

Bolting a pretrained encoder onto a pretrained language model is called late fusion. The alternative is to train one model on text and images from the start (early fusion).

• Fuyu (Adept, 2023) has no separate image encoder: patches are linearly projected straight into the decoder.
• Chameleon (Meta, 2024) turns each 512 × 512 image into 1,024 discrete tokens from a learned codebook of 8,192 image “words”, and trains one model on mixed sequences of text and image tokens.

Hearing: the same trick on sound

Audio is a wave: 16,000 numbers per second at the usual speech sampling rate. The first step turns it into a picture of sound.

Log-mel spectrogram

Chop the sound into short overlapping slices. For each slice, measure how much energy there is in each pitch band, spaced the way human hearing is, on a log scale. Each slice becomes one column of numbers.

For example: Whisper uses 25 ms slices, starting every 10 ms: 100 columns per second, each with 80 numbers (128 in its latest large model).

Whisper (OpenAI, 2022, trained on 680,000 hours of audio) reads 30-second chunks. Its encoder starts with two small convolutions; the second halves the length. Then a Transformer, exactly like the vision encoder, turns the columns into one vector per 20 ms of audio. Audio-language models attach such an encoder to a language model with a projector, often after pooling to shorten the sequence further.

To speak back, some models go one step further. A neural audio codec turns sound into discrete tokens (the codec in Moshi makes 12.5 frames per second, each a small stack of codebook tokens), and the model predicts audio tokens the way it predicts text tokens. Kyutai’s open Moshi (2024) works this way, listening and speaking at the same time.

Let's see the numbers

Work one real image size by hand: 448 × 448 pixels, the tile size used by the InternVL family, with 14-pixel patches.

Pause and predict 448 × 448 pixels, cut into 14 × 14 patches. How many patches, and so how many image tokens before any merging?

1,024 tokens for one small image is a lot, and neighbouring patches often say almost the same thing (look at the sky in the sunset picture). So many models merge each 2 × 2 block of neighbouring patch vectors into one token. The usual method, called pixel shuffle, joins the four vectors end to end into one vector four times as long, and the projector then shrinks that width back down. The image now costs a quarter of the tokens, and every layer whose work grows with the token count does a quarter of the work.

A bigger picture: tiles

An encoder is trained at one size. A tall phone screenshot squeezed to 448 × 448 loses its small text. The common fix is dynamic resolution: cut the image into several 448 × 448 tiles, encode each one, and add one shrunken thumbnail of the whole image so the model still sees the overall layout.

Tiling is one common fix among several. Since Qwen2-VL (2024), several leading open models skip the fixed tile and encode the image at its own size and shape, giving each patch a rotary position for its row and column (2D RoPE) instead of an added position vector; SigLIP 2 (2025) offers a native-aspect-ratio encoder too. The token count still follows the area formula below.

The customer’s one-line complaint is about a dozen text tokens. Their screenshot costs well over a hundred times as many tokens. Images are the expensive part of these tickets, both in how long the model takes to reply and in what you pay for it.

And for sound

A 30-second voice note, Whisper-style: 30 s ÷ 10 ms = 3,000 spectrogram columns. The stride-2 convolution halves that to 1,500 encoder vectors, one per 20 ms. A model that pools by 2 again before the language model gets 750 audio tokens.

The math

Two small formulas cover everything above. First, the token count:

Halve the patch size and N goes up 4 times. Double the image side and N goes up 4 times. Token cost grows with the area, which is why tiling and merging exist.

Second, what one image token is:

It is exactly the same shape as a text token entering a GPT: a vector from the “input side” plus a position vector. The only difference is where the first vector came from: a table lookup for text, a matrix multiply of pixels for images.

What you should remember

1. A model “sees” by turning a picture into a sequence of vectors: cut into P × P patches, flatten, multiply by a learned matrix, add a position. One patch = one token. Sound works the same way: a log-mel spectrogram (100 columns per second) → an encoder (Whisper: one vector per 20 ms) → tokens.
2. Typical open design: vision encoder (a Transformer, often trained contrastively on image-caption pairs) → projector → the language model, which reads image and text tokens as one sequence.
3. Token cost grows with area: N = (H/P)(W/P)/r². 448 × 448 at 14 px is 1,024 patches, 256 tokens after 2 × 2 merging. Tiling adds tokens per tile.
4. Late fusion (bolted-on encoder) is well documented in open models. How closed “natively multimodal” models are built is largely not published.

Where this appears in a real LLM

In this lesson

• 112 × 112 built-in pictures, drawn in code
• Random, untrained projection to at most 12 numbers
• No vision encoder: patch tokens are the end of the story
• Token counts computed with a simplified tiling rule

In a real system

• Images resized, tiled, or processed at their native aspect ratio (Qwen2-VL: one token per 28 × 28 pixels after merging)
• Learned projection to 1,000 or more numbers per token
• A 300-million to 6-billion-parameter vision encoder, then a projector
• Each model family has its own resize, tile and cap rules

Generating pictures, briefly

Reading an image and drawing one are different problems. Two families dominate:

• Diffusion. Start from pure noise and repeatedly remove a little of it, guided by the text prompt, until an image remains. Stable Diffusion and many commercial image generators work this way, usually in a compressed “latent” space rather than on raw pixels.
• Autoregressive image tokens. Turn images into discrete tokens with a learned codebook, then predict them one after another, exactly like text. Chameleon is an open example: one model writes text and images in a single stream.

Since 2025 some chat assistants generate images “natively” rather than by handing off to a separate model, and hybrids exist. Which recipe a given closed product uses is mostly not disclosed. What is established: both families work, and both need huge amounts of paired image-text data.

For the support bot, every screenshot costs about 1,800 tokens before the customer has said a word, so images belong in the token budget alongside the text.