What Is an AI Token?

An AI token is the small chunk of text that a language model actually reads and writes — not quite a word, not quite a single letter, but something in between. When you type a prompt into a chatbot, the model first breaks your text into tokens, processes them as numbers, and then produces new tokens as its reply. Understanding tokens explains why AI tools charge the way they do, why they sometimes seem to “forget” the start of a long conversation, and how to estimate both cost and length before you hit send.

What is an AI token?

A token is a unit of text produced by a process called tokenization. Depending on the model and the language, a token can be as short as a single character or as long as a whole word. Common short words like “the” or “and” are usually a single token, while longer or rarer words get split into smaller pieces. Spaces and punctuation count too, so a comma or a leading space often becomes part of a token.

Models work with tokens rather than raw words because it lets them handle any text — including names, code, typos, and languages they were not explicitly trained on — using a fixed, manageable vocabulary. The model converts each token into a number, does its math, and converts the predicted numbers back into tokens that form the response. If you want the bigger picture of how these systems work, see our explainer on large language models.

Tokens versus words

Tokens and words are related but not the same. As a widely cited rule of thumb for English, one token is roughly four characters, and about 100 tokens is roughly 75 words. Put another way, one token is about three-quarters of a word on average. These are estimates, not exact conversions: the real count depends on the specific text and the model’s tokenizer.

Short, common English words tend to be one token each. Long words, technical terms, URLs, and non-English text often split into several tokens. The table below shows how a few examples typically break down, using the rough ratios above.

Because the ratio shifts with content, the only way to get an exact count is to run the text through the model’s own tokenizer tool. The four-characters estimate is close enough for planning but should not be treated as precise, and it does not transfer cleanly between different providers’ models.

How a sentence becomes tokens

Imagine the sentence “I love tokenization!” A typical English tokenizer might split it into something like “I”, ” love”, ” token”, “ization”, and “!” — five tokens from three words and a punctuation mark. Notice two things: the space before a word is usually attached to that word rather than standing on its own, and an uncommon word like “tokenization” is broken into familiar sub-word pieces the model has seen many times before.

This sub-word approach is what lets a model read words it has never encountered as whole units. A brand-new product name or a rare technical term is simply assembled from smaller known pieces, which is also why unusual text tends to use more tokens than plain, everyday prose. The practical upshot is that you cannot reliably judge token count by eye: short-looking text full of rare words or symbols can cost more than you expect.

Why tokens matter: cost

Most commercial AI APIs bill by the token. You typically pay one rate for input tokens (your prompt plus any context you send) and another, often higher, rate for output tokens (the model’s response). That means the length of both your question and the answer affects what you pay, and a verbose prompt with a long reply costs more than a tight one that gets to the point.

For anyone building on these tools, token math is budgeting. Summarizing or trimming context before you send it, and asking for concise outputs when you do not need long ones, can noticeably lower your bill at scale. This is one reason prompt design matters so much; our guide to writing better prompts covers techniques that keep requests efficient without losing quality.

Why tokens matter: the context window

The context window is the maximum number of tokens a model can consider at once — your prompt, the conversation history, any documents you paste, and the response, all counted together. Every model has a limit. When a conversation grows past that limit, the earliest tokens fall out of view, which is why a long chat can seem to “forget” something you said near the beginning.

Context windows have grown dramatically, with some current models handling hundreds of thousands of tokens or more, but there is always a ceiling, and filling it carries both cost and performance consequences. Managing what goes into that window — keeping the relevant material and dropping the rest — is a core skill when working with generative AI and with autonomous systems like an AI agent that may loop through many steps, each one spending tokens.

Tokens in practice

A few practical takeaways help you work with tokens day to day:

• Estimate before you send: for English, divide your character count by about four to approximate the number of tokens.
• Budget both directions: you pay for input and output, so cap response length when you can.
• Mind long documents: pasting a big file can consume a large share of the context window on its own.
• Non-English costs more: many languages tokenize into more tokens per word than English, raising both cost and context use.
• Counts vary by model: the same text produces different token counts across providers, so check the tokenizer for the model you actually use.

Tokens are the hidden currency of modern AI: they determine what you pay, how much the model can “remember” at once, and how long a response can run. Once you start thinking in tokens rather than words, cost estimates and context limits stop feeling mysterious and become something you can plan around. Explore more foundational concepts in our AI section.