AI Reference 051 Large Language Models Wikipedia guided Primary sources linked

Tokenization

Definition

Tokenization converts text into token identifiers a model can process. Model families use different tokenizers, so the same text can consume different numbers of tokens in different systems.

What is tokenization in an LLM?

Tokenization is the conversion of raw text into a sequence of token identifiers. The tokenizer uses a fixed vocabulary learned or designed before model training. A word may stay whole if it is common or be split into several subword pieces if it is rare.

Byte-pair encoding and related methods

Many LLM tokenizers are descended from byte-pair encoding or similar subword algorithms. Frequent character sequences become vocabulary entries, allowing common text to be represented compactly while still preserving a way to encode unusual strings.

Why tokenization changes model behavior

Token boundaries affect sequence length, cost and sometimes reasoning difficulty. Numbers, unusual spelling or non-Latin scripts can split differently depending on the tokenizer. Tokenization is entirely separate from financial asset tokenization, which refers to representing ownership claims as digital tokens on a ledger.

Related terms, defined

Large Language Models Context Windows A context window is the amount of tokenized information a model can consider in one working sequence or request. Larger context increases capacity for long inputs but does not guarantee perfect retrieval or reasoning over every detail. Large Language Models Embeddings in Modern AI Embeddings are dense numerical vectors used to represent text, images or other objects in spaces where distance or similarity can be computed. Large Language Models Inference Inference is the phase in which a trained model is run to make predictions or generate outputs. LLM inference usually produces tokens sequentially. Large Language Models Temperature and Sampling Temperature changes the probability distribution used during sampling. Lower values concentrate probability more strongly; higher values make less likely alternatives easier to sample. Large Language Models AI Hallucinations A hallucination is generated content that is false, unsupported or inconsistent with the evidence available to the system. Language generation optimizes likely sequences, not guaranteed factual truth, so high-stakes factual work needs grounding and verification.

Reference guide and primary sources

Wikipedia is used here as a terminology and history reference guide. Current model versions, institutional statistics and product-specific claims are also linked to first-party or institutional sources because those details can change faster than encyclopedia articles.

← Tokens in AI Context Windows →