AI development tools · 10/14/2026 · Updated 10/15/2026

What Are Tokens in LLMs? Tokens vs Words and Context Windows

A token is the unit of text a large language model (LLM) actually reads and writes: a common word, a piece of a longer word, a punctuation mark or a few digits, each stored as an ID number. In ordinary English, one token averages about 4 characters, or three-quarters of a word, so 100 tokens is roughly 75 words. Korean, Japanese and Chinese usually fit fewer characters into each token, and every tokenizer counts differently. Prices, rate limits and the context window, which must hold your prompt and the model's answer together, are all measured in tokens.

The sentence “Tokenization isn't magic: words like Morrow split into pieces!” split by OpenAI's current tokenizer into 13 color-coded tokens with their ID numbers: Token, ization, isn't, magic, colon, words, like, M, orrow, split, into, pieces, exclamation mark. — Figure 1. One sentence as OpenAI's current tokenizer splits it: 62 characters and 9 words become 13 tokens. Common words keep their leading space, while “Tokenization” and the rare name “Morrow” are split into pieces.

How a tokenizer turns text into tokens

A model never sees letters or words directly. A tokenizer cuts the text into pieces from a fixed vocabulary and swaps each piece for its ID; the model predicts the next ID, one token at a time, and the tokenizer turns the IDs back into text. Figure 1 shows four common patterns:

Common words are one token, space included. As OpenAI's documentation notes, the first token of a word usually starts with a space, so “token”, “ token” and “ Token” have three different IDs.
Long or rare words split into pieces. “Tokenization” becomes “Token” + “ization”, and “Morrow” becomes “ M” + “orrow”.
Punctuation usually stands alone, like the colon and the exclamation mark.
Numbers break into chunks: the date 2026-10-15 takes 6 tokens.

Byte pair encoding in three merges

The most popular way to build a tokenizer is byte pair encoding (BPE). Training starts from single characters, or the 256 possible byte values, and repeatedly merges the pair of neighbors that appears most often in a large body of text, adding each merge to the vocabulary. In Figure 2, “u” + “g” is merged first because it occurs 20 times, then “u” + “n” (16 times), then “h” + “ug” (15 times). Real tokenizers continue until the vocabulary reaches a target size; GPT-2 used 50,000 merges on top of 256 byte tokens. Because the bottom layer is bytes, any text can be encoded, even an emoji. Rare text just costs more tokens.

Byte pair encoding on five toy words (hug ×10, pug ×5, pun ×12, bun ×4, hugs ×5): starting from single characters, merge 1 joins u and g (20 times), merge 2 joins u and n (16 times), merge 3 joins h and ug (15 times). — Figure 2. Byte pair encoding on a toy word list: each step merges the most frequent adjacent pair, so after three merges “hugs” is “hug” + “s”.

How many tokens per word? Rules of thumb

For quick estimates, OpenAI and Google's Gemini API documentation publish rough ratios for English text:

Rule of thumb (English)	Approximate value	Stated by
Characters per token	about 4	OpenAI, Google
Words per token	about 0.75	OpenAI
Words per 100 tokens	about 75 (OpenAI); 60–80 (Google)	OpenAI, Google
Tokens per 1,000 words	about 1,333 (1,000 ÷ 0.75)	Calculated from OpenAI's ratio

Test them on Figure 1: 62 characters ÷ 4 predicts 15.5 tokens, 9 words ÷ 0.75 predicts 12, and the real count is 13. That is close enough for budgeting, but the ratios drift for code, URLs, numbers and non-English text.

Why Korean, Japanese and Chinese count differently

A tokenizer's vocabulary reflects the text it was trained on. Languages that were less common in that text get fewer long merged pieces, so their sentences break into more, shorter tokens. Byte-level tokenizers add a twist: a Korean syllable or a Chinese or Japanese character takes 3 bytes in UTF-8, and a character missing from the vocabulary can split into two or three byte fragments, which OpenAI's Tokenizer displays as �.

OpenAI's GPT-4o announcement counted one greeting in 20 languages with its previous tokenizer and its new one; Figure 3 shows four of them. With the older tokenizer, the Korean version needed 45 tokens against 27 for English, about 1.7 times as many. The new tokenizer cut Korean to 27, Japanese from 37 to 26 and Chinese from 34 to 24, close to English's 24.

Bar chart of tokens for the same greeting. Older OpenAI tokenizer: English 27, Korean 45, Japanese 37, Chinese 34. GPT-4o tokenizer: English 24, Korean 27, Japanese 26, Chinese 24. — Figure 3. Tokens needed for the same greeting in four languages under OpenAI's older tokenizer and the GPT-4o tokenizer. Character counts include spaces and punctuation.

Two lessons follow. First, the English “4 characters per token” rule does not transfer: in the GPT-4o counts, spaces and punctuation included, Korean averages about 2.0 characters per token, Japanese 1.7 and Chinese 1.4, against 3.4 for English. Second, a count belongs to a tokenizer, not to the text. Anthropic's token counting guide, for instance, says the tokenizer introduced with Claude Opus 4.7 produces roughly 30 percent more tokens for the same input than earlier Claude models.

What is a context window?

The context window is the maximum number of tokens a model can handle in one request. It is one shared budget: Anthropic defines it as all the text the model can reference, “including the response itself,” and OpenAI says the prompt and the output together must fit within the model's maximum context length. The window holds:

the system prompt and any tool definitions;
the conversation so far, because a chat resends earlier turns with every new message;
attached documents, images and PDFs, which are converted to tokens too;
your new message;
the answer being generated, including reasoning (“thinking”) tokens on models that use them.

A 100,000-token context window drawn as a bar: system prompt 5,000, chat history 30,000, attached document 40,000 and new message 3,000 make 78,000 input tokens; the answer takes 10,000 output tokens and 12,000 are unused. Three cards show what happens when the total will not fit. — Figure 4. One context window holds the system prompt, chat history, attachments, the new message and the answer. Illustrative numbers for a 100,000-token window.

When the total will not fit, one of three things happens:

The request is rejected. If the input alone exceeds the window, the API returns an error; Claude's, for example, reports that the prompt is too long.
The answer is cut off. If the reply runs out of room, or reaches the output cap you set (often called max_tokens), generation simply stops.
Old turns are dropped or summarized. Chat apps may trim the earliest messages or compress them into a summary.

A bigger window is not automatically better. Anthropic notes that accuracy and recall degrade as the token count grows, so a focused prompt often beats an overstuffed one.

Why token counts matter: cost, speed and limits

Cost. API usage is billed per token, usually quoted per million tokens, with separate rates for input and output; output tokens typically cost several times more. In an API-based chat, the whole history is sent, and billed as input, again on every turn. Prompt caching lowers the price of repeated text, but cached tokens still occupy the window.
Speed. Models write one token at a time, so a 2,000-token answer takes much longer than a 200-token one.
Limits. Rate limits, usage quotas and output caps are counted in tokens, and a reply that hits its cap ends mid-sentence.

How to count tokens exactly

For a quick look, paste text into OpenAI's public Tokenizer page, which highlights every token and shows the total.
In code for OpenAI models, use the open-source tiktoken library; tiktoken.encoding_for_model() picks the right encoding for a model.
For Claude, call the free token counting endpoint, whose result Anthropic describes as an estimate. For Gemini, call count_tokens.
After a real request, read the usage field in the response, which reports the input and output tokens actually used.

FAQ

Do spaces and punctuation count as tokens?

Yes. A single space usually rides along with the next word, but extra spaces, line breaks and punctuation can add tokens. In OpenAI's current tokenizer, “Hello world” is 2 tokens, while the same two words separated by two spaces take 3.

Why does a long chat forget what I said at the start?

The conversation has outgrown the context window, so the app has dropped or summarized the earliest turns. Starting a new chat with a short summary of what matters usually works better.

Do ChatGPT, Claude and Gemini count tokens the same way?

No. Each model family uses its own tokenizer, so the same text gives different counts. To compare, measure the same text with each model's own tool.

Sources: OpenAI API documentation, Key concepts (Tokens); OpenAI, Hello GPT-4o (Language tokenization); Anthropic Claude documentation, Context windows; Hugging Face Transformers documentation, Tokenization algorithms.