What Is a Token?

Also called tokenization

Short Answer

A token is the unit of text a language model reads and writes: a whole word, part of a word or a punctuation mark. In English, one token is on average about four characters, or roughly three quarters of a word. Model limits and API prices are counted in tokens.

  • Taught in lesson 8

Why It Matters

Tokens are how language models measure everything: what they can read, what they can write and what they cost. Anyone building with AI needs to estimate tokens to predict cost, stay within limits and understand why the same text costs more in some languages than others.

How It Works

Before a model sees your text, a tokenizer splits it into tokens and turns each one into a number. Common words are often a single token; rare words, names and many non-English words split into several tokens, which makes them more expensive to process.

Tokens matter for two practical reasons: the context window, the maximum number of tokens a model can handle at once, and cost, because providers charge per million input and output tokens.

A tokenizer is built from a large sample of text by finding pieces that appear often. Frequent words become single tokens, rarer words are split into parts, and numbers and code can split in surprising ways. That is why models can struggle with tasks such as counting letters in a word: they see tokens, not letters. Different model families use different tokenizers, so token counts for the same text vary between them.

Token in Practice

Everyday Example

The word "unbelievable" may be split into tokens such as "un", "believ" and "able". A one-page email is usually several hundred tokens.

Industry Example: Pharma

Long drug names and gene symbols often split into many tokens, so a pharma team estimating the cost of processing thousands of documents counts tokens, not pages.

Where You Meet It

  • API pricing, quoted per million tokens.
  • Context window limits.
  • Usage dashboards of AI tools.
  • Errors when a request is longer than a model allows.

Key Ideas Around It

Tokenizer
The tool that splits text into tokens.
Input and output tokens
Tokens you send and tokens the model writes, often priced differently.
Context window
The total tokens a model can handle at once.
Vocabulary
The full set of tokens a model knows.

A Common Mix-Up

A token is not a word. In English a token is about three quarters of a word on average, and in many other languages one word often takes several tokens.

Check YourselfWhy can the same message cost more in one language than another?

Because tokenizers split some languages into more tokens per word, and cost is counted in tokens.

Learn It on ai2hired

S0 · #8Tokens, Context Windows and Model Families

Coming in Season 0, Start Here. Every ai2hired lesson is free on YouTube.

Questions People Ask

How many tokens is a word?

In English, about 1.3 tokens per word on average. The exact number depends on the model and its tokenizer.

Why do output tokens cost more than input tokens?

The model generates output tokens one at a time, which takes more computing work than reading the input, so providers usually price them higher.

Learn AI From Zero to Hired

Free lessons on YouTube, each one built from what real AI job ads ask for.