Tokens and Context Windows Explained: How AI Reads Your Text

Language models do not read words, they read tokens. What a token is, how byte pair encoding chops text into pieces, why the same sentence costs more tokens in some languages, what the context window really holds, and why a bigger window does not mean the model pays equal attention to everything in it.

0:28 What a token is

Models process text as tokens, pieces that can be a character, part of a word, a whole word or punctuation; in English one token is roughly three quarters of a word.

Sources: OpenAI Help Center (original) · GitHub (original)

1:50 How text is cut into pieces

Byte pair encoding starts from single characters and keeps merging the most frequent neighboring pair, so common words stay whole and rare words split into known pieces.

Sources: Hugging Face Transformers docs (original) · arXiv (original) · GitHub (original)

3:38 Why some text costs more tokens

Token counts depend on the model, its encoding and the language, so the same idea can take a different number of tokens.

Sources: OpenAI Help Center (original) · Hugging Face Transformers docs (original)

4:54 What the context window holds

The context window is the model's working memory: everything it can refer to while answering, including its own answer.

Sources: Claude Docs (original) · OpenAI Help Center (original)

6:08 Bigger is not always better

Accuracy and recall can degrade as the context grows, and models often use information at the start or end of a long input better than information in the middle.

Sources: Claude Docs (original) · arXiv (original) · OpenAI Help Center (original)

Follow the show in your podcast app to get each new episode as soon as it is out.

Produced by Juan Pablo.