AI Reference 050 Large Language Models Wikipedia guided Primary sources linked

Tokens in AI

Definition

Tokens are the discrete units processed by a language model. Depending on the tokenizer, a token can represent a word, part of a word, punctuation, whitespace patterns or another text fragment.

What is a token?

A token is one discrete unit of input or output used by a language model. Tokens are produced by a tokenizer and may correspond to a whole word, part of a word, punctuation, whitespace or bytes, depending on the tokenizer.

Why models use tokens instead of words

Natural language contains too many possible words, spellings, names and character combinations to assign every possible word a unique learned entry efficiently. Subword tokenization gives models a manageable vocabulary while still allowing uncommon words to be represented as sequences of smaller pieces.

Why token counts matter

Model context limits, API billing and generation speed are often measured in tokens. The same sentence can produce different token counts in different model families because tokenizers differ. A token is therefore not a fixed number of characters or a fixed fraction of a word.

Related terms, defined

Large Language Models Tokenization Tokenization converts text into token identifiers a model can process. Model families use different tokenizers, so the same text can consume different numbers of tokens in different systems. Large Language Models Context Windows A context window is the amount of tokenized information a model can consider in one working sequence or request. Larger context increases capacity for long inputs but does not guarantee perfect retrieval or reasoning over every detail. Large Language Models Embeddings in Modern AI Embeddings are dense numerical vectors used to represent text, images or other objects in spaces where distance or similarity can be computed. Large Language Models Inference Inference is the phase in which a trained model is run to make predictions or generate outputs. LLM inference usually produces tokens sequentially. Large Language Models Temperature and Sampling Temperature changes the probability distribution used during sampling. Lower values concentrate probability more strongly; higher values make less likely alternatives easier to sample.

Reference guide and primary sources

Wikipedia is used here as a terminology and history reference guide. Current model versions, institutional statistics and product-specific claims are also linked to first-party or institutional sources because those details can change faster than encyclopedia articles.

← RLHF and Preference Training Tokenization →