AI Reference 041 Neural Networks & Deep Learning Wikipedia guided Primary sources linked

The Transformer Architecture

Definition

The Transformer was introduced in the 2017 paper “Attention Is All You Need.” It replaced recurrent sequence processing with attention-based layers, enabled highly parallel training and became the architectural basis of most modern large language models.

What is a transformer?

A transformer is a neural-network architecture built around self-attention and feed-forward layers. The original architecture was introduced in the 2017 paper 'Attention Is All You Need' for machine translation.

How a transformer processes tokens

Tokens are converted into vectors and combined with positional information. Self-attention lets every token construct a context-dependent representation from other allowed tokens. Feed-forward networks then transform each position, and the process repeats through many layers.

Why transformers became the basis of LLMs

Unlike recurrent networks, transformers can process many sequence positions in parallel during training. That property scaled well on modern hardware. Decoder-only transformer variants became the standard architecture behind most large autoregressive language models.

Related terms, defined

Neural Networks & Deep Learning Encoder-Decoder Models Encoder-decoder architectures turn an input into internal representations and then generate an output sequence or structure conditioned on those representations. Neural Networks & Deep Learning Self-Supervised Learning Supervised learning trains a model from examples paired with known labels or target values. Classification and regression are common supervised tasks. Neural Networks & Deep Learning Foundation Models Foundation models are trained on broad data at scale and then adapted or prompted for many downstream tasks. Neural Networks & Deep Learning Multimodal AI Multimodal AI processes or generates multiple data types such as text, images, audio or video in one coordinated system. Neural Networks & Deep Learning Artificial Neural Networks Artificial neural networks are parameterized models built from layers of connected computational units. Training adjusts the parameters to reduce an objective function.

Reference guide and primary sources

Wikipedia is used here as a terminology and history reference guide. Current model versions, institutional statistics and product-specific claims are also linked to first-party or institutional sources because those details can change faster than encyclopedia articles.

← Attention Mechanisms Encoder-Decoder Models →