Models do not read letters and they do not read words. They read tokens — chunks of text somewhere between the two — and almost every strange thing a model does with spelling, numbers or foreign languages traces back to that.
Second in a series on how language models actually work.
Text is cut into pieces before the model sees it
A tokeniser splits text into units drawn from a fixed vocabulary, typically 50,000 to 200,000 entries. Common words are single tokens. Rare words break into fragments. Spaces usually attach to the front of the following word.
The rule is frequency, not meaning: pieces that appeared often in the training text became tokens, and everything else is assembled from smaller ones.
Why models miscount letters
Ask a model how many rs are in strawberry and it may well get it wrong. This is not a reasoning failure. By the time the model sees the word it has become two or three numeric ids, and the individual letters are simply not present in the input. Asking it to count them is like asking someone to count the letters in a word they only ever heard spoken.
The same mechanism explains why models are shaky at reversing strings, at rhyming in some languages, and at character-level edits. Where the task genuinely needs letters, the fix is to give it letters — space them out, or hand the job to code, which is what tool use is for.
Tokens are the unit of cost and of limits
Pricing, rate limits and context windows are all measured in tokens, so it is worth knowing the rough exchange rate. English prose runs about 0.75 words per token. Code is denser — punctuation and indentation each cost something. Languages that do not use Latin script frequently cost several times more per character, because the vocabulary was built mostly from English text.
That last point has a real consequence: the same document can cost three or four times as much to process in one language as another, and it eats the context window proportionally faster. If you are building for a non-English audience, measure it rather than assuming parity.
Embeddings: from ids to meaning
A token id is arbitrary — 4589 is not more or less than 8022 in any meaningful sense. The first thing the model does is replace each id with a vector, a long list of numbers, learned during training.
These vectors are where similarity lives. Tokens used in similar contexts end up with similar vectors, which is why arithmetic on them is famously almost sensible, and why retrieval works: embed a question and a paragraph, compare the vectors, and closeness is a decent proxy for relevance.
Worth being precise about a common confusion. The embeddings inside the model are per-token. The embeddings you use to search documents are usually one vector for a whole passage, produced by a related but separately trained model. Same idea, different tool.
Practical consequences
Three things follow that are worth remembering:
- Odd spacing costs money. Unusual whitespace and formatting fragment into more tokens than clean text. Long JSON with heavy indentation is measurably more expensive than compact JSON.
- Rare identifiers are expensive. A UUID can consume a dozen tokens. Passing hundreds of them wastes context you may need later.
- Trailing spaces matter. Because spaces bind to the following word, a prompt ending in a space can force an awkward tokenisation of whatever comes next. Ending cleanly is a free improvement.
What to take away
The model reads a sequence of numbers standing for text fragments, then swaps each for a learned vector. Letters do not survive that trip. Once you have that picture, the spelling failures stop being mysterious and start being predictable — which means you can design around them.
Next in the series: attention, and how a model decides which earlier words matter.