AI Reference 047 Large Language Models Wikipedia guided Primary sources linked

LLM Pretraining

Definition

LLM pretraining exposes a model to large datasets and optimizes a predictive objective, commonly next-token prediction, before later post-training or adaptation.

What happens during pretraining?

Pretraining is the large-scale training stage that creates the base language model. For an autoregressive LLM, text is tokenized and the network is trained to predict the next token from the tokens that came before it.

Where the training signal comes from

The training target is generated automatically from the text itself: every token in a sequence can become a target for the preceding context. That is why modern LLMs can learn from enormous unlabeled corpora without human annotators writing a correct answer for every example.

What pretraining does not provide

A pretrained model may complete text but still be difficult to control as an assistant. Instruction tuning, preference training, safety training and tool integration are later stages. Pretraining also does not make the model a guaranteed source of factual truth.

Related terms, defined

Large Language Models Fine-Tuning Language Models Fine-tuning continues training a pretrained model on a narrower dataset or objective so it performs better for a particular domain, task or behavior. Large Language Models RLHF and Preference Training Reinforcement learning from human feedback uses preference information from people to train or guide models toward preferred behavior; modern post-training also uses related preference-optimization techniques. Large Language Models Tokens in AI Tokens are the discrete units processed by a language model. Depending on the tokenizer, a token can represent a word, part of a word, punctuation, whitespace patterns or another text fragment. Large Language Models Tokenization Tokenization converts text into token identifiers a model can process. Model families use different tokenizers, so the same text can consume different numbers of tokens in different systems. Large Language Models Context Windows A context window is the amount of tokenized information a model can consider in one working sequence or request. Larger context increases capacity for long inputs but does not guarantee perfect retrieval or reasoning over every detail.

Reference guide and primary sources

Wikipedia is used here as a terminology and history reference guide. Current model versions, institutional statistics and product-specific claims are also linked to first-party or institutional sources because those details can change faster than encyclopedia articles.

← What Is a Large Language Model? Fine-Tuning Language Models →