What Is a Token in AI?
How LLM Tokens Work

A token in AI is a small piece of text (a whole word, part of a word, a punctuation mark or a space-plus-word fragment) that a language model uses as its basic unit. A tokenizer splits your text into these pieces and maps each one to a numeric token ID. The model turns every ID into a vector (an embedding), processes the sequence, and predicts the next token, one at a time. Token counts also set practical limits: how much fits in a request, and, for many APIs, what the request is metered at.

A token is a unit from a model’s vocabulary, the fixed list of pieces its tokenizer knows. OpenAI’s help center describes tokens as units that “can represent a character, part of a word, a whole word, or punctuation,” and notes that spaces also affect how text is divided.

Two consequences follow straight away:

• A token is not necessarily a word. A common word may be one token; a rare word, a name or a code identifier may be several.
• Tokens are model-specific. Different models use different vocabularies and rules, so the same sentence can produce a different number of tokens, and different token IDs, depending on the model.

The exact pipeline depends on the tokenizer, but most LLM tokenizers follow the same broad shape. Hugging Face’s tokenizer documentation describes it as normalization, pre-tokenization, the tokenization model itself, and post-processing; decoding reverses it.

Step 1: Prepare the text

The tokenizer may normalize the input (for example, handling Unicode or accents in a consistent way) and pre-tokenize it into rough chunks, often around spaces and punctuation. How it treats spaces, capitals and symbols is implementation-specific. OpenAI’s guide points out that “red”, “Red” and ” red” (with a leading space) are not identical text, and an encoding can represent them differently.

Step 2: Split the text into vocabulary pieces

Next, each chunk is matched to pieces in the vocabulary. Common words often stay whole; less common ones are split into subwords. Hugging Face’s example: annoyingly might become ["annoying", "ly"] or ["annoy", "ing", "ly"], depending on the vocabulary.

Step 3: Map each piece to a token ID

Every vocabulary entry has a number. The tokenizer outputs a list of these IDs, plus any special tokens the model’s input format needs (for example, markers for the start of a message or the end of a text). The IDs are just lookup numbers; their meaning depends entirely on the encoding, so an example ID from one model doesn’t apply to another.

Step 4: Turn IDs into embeddings

Inside the model, each ID selects a row from an embedding matrix: a learned vector of numbers that represents that token. The model works on these vectors, not on letters. Our embeddings lesson shows what those vectors capture, and how a transformer works follows them through attention and feed-forward layers.

Step 5: Predict the next token

For text generation, the model scores every token in its vocabulary as a possible next piece. A decoding rule picks one (this is where settings like temperature come in), the chosen token is appended, and the process repeats.

Step 6: Decode back to text

The tokenizer’s decoder joins the generated pieces back into readable characters and words. That’s the round trip: text → pieces → IDs → embeddings → predicted IDs → text.

Why not just use one token per word? Hugging Face explains the trade-off:

• Word-level vocabularies get huge, since every variation (“love”, “loving”, “loved”, “lovingly”) needs its own entry, and unseen words fall back to an “unknown” token.
• Character-level vocabularies are tiny and cover everything, but sequences get much longer and each piece carries little meaning.
• Subword methods sit in between: common words stay whole, rare words are assembled from known pieces.

The three subword algorithms Hugging Face Transformers supports:

A tiny BPE walkthrough

Hugging Face uses a toy corpus of the words hug, pug, pun, bun, hugs:

1. Start with single characters: b g h n p s u.
2. The pair u+g appears most often (in hug, pug, hugs), so merge it into ug.
3. Next, u+n (in pun, bun) merges into un.
4. Keep merging until the vocabulary hits its target size.

Byte-level BPE goes one step further: it uses the 256 possible byte values as the base, so any text can be encoded without an “unknown” token. The docs give GPT-2 as an example, with a vocabulary of 50,257 (256 bytes + 50,000 merges + one end-of-text token).

SentencePiece is a library that applies BPE or Unigram directly to raw text, treating the space as a symbol (▁). It’s useful for languages such as Chinese and Japanese that don’t separate words with spaces.

There’s no exact conversion, but OpenAI publishes rough estimates for English text:

• 1 token ≈ 4 characters
• 1 token ≈ ¾ of a word
• 100 tokens ≈ 75 words

Treat these as planning shortcuts, not counts. OpenAI notes that other languages can have different character-to-token ratios, and capitalization, spacing, punctuation and code all change the split. If you need a real number, run the text through the tokenizer for the model you’ll actually use.

The context window is measured in tokens

A model can only work with a limited number of tokens per request: its context window. Your instructions, the conversation history, any pasted documents, and the model’s answer all draw from that budget, and models also have a separate output limit. OpenAI’s guide adds that reasoning models use internal reasoning tokens that count toward output too. Read what a context window in AI is for how that budget works and how to manage it.

When a document is too big to fit, a common approach is to retrieve only the relevant passages instead of pasting everything; that’s the idea behind retrieval-augmented generation (RAG).

Many APIs meter usage in tokens

Many LLM APIs charge per token, and OpenAI’s documentation notes that input, cached input and output tokens can be priced differently, with rates that depend on the model. Because models tokenize the same text differently and produce different amounts of output, a lower per-token rate doesn’t automatically mean a cheaper task. Check each provider’s current pricing page instead of relying on figures copied from elsewhere.

Counting tokens accurately

1. Pick the right tokenizer. Use the one tied to the model you’re calling (for OpenAI models, the interactive Tokenizer page or the tiktoken library).
2. Count the whole request. Message structure, tool schemas, images and files can add tokens a plain-text count misses.
3. Reserve room for the answer. Don’t spend the whole limit on the prompt.
4. Check actual usage. API responses report token usage after each call.

• OpenAI Help Center, Understanding and counting tokens: token definition, English estimates, spaces and capitalization, input/output/cached/reasoning tokens, counting and pricing notes.
• Hugging Face Transformers, Tokenization algorithms: BPE, byte-level BPE, WordPiece, Unigram, SentencePiece, word vs character tokenization, toy examples.
• Hugging Face Tokenizers, The tokenization pipeline: normalization, pre-tokenization, model, post-processing, decoding (carried over from live page).
• OpenAI, Tokenizer (carried over from live page).

Examples and rough estimates above are tied to their cited tokenizer documentation; model families do not all tokenize the same way.