AI Reference 060 Large Language Models Wikipedia guided Primary sources linked

Mixture-of-Experts Models

Definition

A mixture-of-experts model contains multiple expert subnetworks and routes each token or input through only a subset of them. This can increase total parameter capacity without activating every parameter on every forward pass.

What is a mixture-of-experts model?

A mixture-of-experts, or MoE, model contains several expert subnetworks and a router that decides which experts process each token or input. Only a subset of the total parameters may be active for one forward pass.

Why use experts?

MoE can increase total model capacity without paying the full computation cost of activating every parameter for every token. The router learns to direct different inputs toward different experts.

Examples in modern LLMs

MoE designs appear in model families such as Mixtral, DeepSeek and Llama 4. A model can therefore have a very large total parameter count while using a much smaller number of active parameters per token. Total parameters and active parameters are different quantities.

Related terms, defined

Large Language Models What Is a Large Language Model? A large language model is a statistical model trained on large collections of text or multimodal data to predict and generate token sequences. Modern LLMs are usually transformer-based and are adapted to conversation, coding, tool use and other tasks through post-training and application-level systems. Large Language Models LLM Pretraining LLM pretraining exposes a model to large datasets and optimizes a predictive objective, commonly next-token prediction, before later post-training or adaptation. Large Language Models Fine-Tuning Language Models Fine-tuning continues training a pretrained model on a narrower dataset or objective so it performs better for a particular domain, task or behavior. Large Language Models RLHF and Preference Training Reinforcement learning from human feedback uses preference information from people to train or guide models toward preferred behavior; modern post-training also uses related preference-optimization techniques. Large Language Models Tokens in AI Tokens are the discrete units processed by a language model. Depending on the tokenizer, a token can represent a word, part of a word, punctuation, whitespace patterns or another text fragment.

Reference guide and primary sources

Wikipedia is used here as a terminology and history reference guide. Current model versions, institutional statistics and product-specific claims are also linked to first-party or institutional sources because those details can change faster than encyclopedia articles.

← AI Agents The GPT Model Family →