AI Reference 040 Neural Networks & Deep Learning Wikipedia guided Primary sources linked

Attention Mechanisms

Definition

Attention computes context-dependent weighted combinations of representations, allowing a model to emphasize information relevant to the current prediction.

What is attention in machine learning?

Attention is a mechanism that lets a model compute weighted combinations of other representations. The weights depend on the current input, so the model can emphasize information that appears most relevant to the current prediction.

Queries, keys and values

In the form used by transformers, each token is projected into query, key and value vectors. Similarity between a query and keys determines attention weights; those weights are used to combine value vectors.

Why attention changed sequence modeling

Earlier sequence models processed tokens recurrently. Attention made it easier to connect distant positions directly. Multi-head attention lets different sets of learned projections capture different relationships in parallel, which became a defining feature of transformers.

Related terms, defined

Neural Networks & Deep Learning The Transformer Architecture The Transformer was introduced in the 2017 paper “Attention Is All You Need.” It replaced recurrent sequence processing with attention-based layers, enabled highly parallel training and became the architectural basis of most modern large language models. Neural Networks & Deep Learning Encoder-Decoder Models Encoder-decoder architectures turn an input into internal representations and then generate an output sequence or structure conditioned on those representations. Neural Networks & Deep Learning Self-Supervised Learning Supervised learning trains a model from examples paired with known labels or target values. Classification and regression are common supervised tasks. Neural Networks & Deep Learning Foundation Models Foundation models are trained on broad data at scale and then adapted or prompted for many downstream tasks. Neural Networks & Deep Learning Multimodal AI Multimodal AI processes or generates multiple data types such as text, images, audio or video in one coordinated system.

Reference guide and primary sources

Wikipedia is used here as a terminology and history reference guide. Current model versions, institutional statistics and product-specific claims are also linked to first-party or institutional sources because those details can change faster than encyclopedia articles.

← Deep Learning The Transformer Architecture →