AI Reference 054
Large Language Models
Wikipedia guided
Primary sources linked
Inference
Definition
Inference is the phase in which a trained model is run to make predictions or generate outputs. LLM inference usually produces tokens sequentially.
MODEL INPUT / TOKEN SEQUENCEinputtokencontextnext
What is inference?
Inference is the stage in which a trained model is used to produce predictions or generated output. Training changes parameters; inference normally keeps those parameters fixed and computes an output from new input.
How LLM inference works
For an autoregressive LLM, the model processes the current token sequence and produces probabilities for the next token. A decoding rule chooses a token, that token is appended to the sequence, and the process repeats until generation stops.
Why inference is expensive
Large models require large amounts of memory bandwidth and computation for every generated token. Techniques such as quantization, batching, caching, speculative decoding and mixture-of-experts routing aim to reduce latency and cost without retraining the entire model.
Related terms, defined
These nearby terms are linked because they name distinct concepts that are easy to confuse with this page's subject.
Large Language Models
Temperature and Sampling
Temperature changes the probability distribution used during sampling. Lower values concentrate probability more strongly; higher values make less likely alternatives easier to sample.
Large Language Models
AI Hallucinations
A hallucination is generated content that is false, unsupported or inconsistent with the evidence available to the system. Language generation optimizes likely sequences, not guaranteed factual truth, so high-stakes factual work needs grounding and verification.
Large Language Models
Retrieval-Augmented Generation
Retrieval-augmented generation combines a generative model with retrieval from an external collection such as documents, databases or search results. Retrieved material is inserted into the model's context so the answer can be grounded in information outside the model's parameters.
Large Language Models
Function Calling and Tool Use
Function calling and tool use let a model request structured actions such as querying a database, calling an API or using search, with the surrounding application executing the action.
Large Language Models
AI Agents
An AI agent is a system in which a model can select or plan actions over multiple steps, usually with tools, memory and feedback from an environment. The language model is only one part of the agent; orchestration, permissions, tools and stopping conditions are equally important.
Reference guide and primary sources
Wikipedia is used here as a terminology and history reference guide. Current model versions, institutional statistics and product-specific claims are also linked to first-party or institutional sources because those details can change faster than encyclopedia articles.