AI Reference 038 Neural Networks & Deep Learning Wikipedia guided Primary sources linked

GPUs and the Rise of Deep Learning

Definition

GPUs perform many numerical operations in parallel, which fits the matrix and tensor workloads used in neural-network training and inference.

What is a GPU?

A graphics processing unit is a processor designed to perform many calculations in parallel. GPUs were created for graphics workloads, where large numbers of pixels and vertices require similar operations.

Why neural networks fit GPUs

Neural networks rely heavily on matrix multiplication and tensor operations that can be split across many parallel computing units. That made GPUs much faster than general-purpose CPUs for many training workloads.

Why hardware changed AI progress

Deep learning did not take off because of one algorithm alone. Better GPUs, large datasets and improved training techniques arrived together. AlexNet's 2012 ImageNet system famously used GPUs, and later AI accelerators expanded the same basic trend toward specialized parallel computation.

Related terms, defined

Neural Networks & Deep Learning Deep Learning Deep learning uses neural networks with multiple layers of learned representations. Its modern rise combined larger datasets, better training methods and parallel hardware. Neural Networks & Deep Learning Attention Mechanisms Attention computes context-dependent weighted combinations of representations, allowing a model to emphasize information relevant to the current prediction. Neural Networks & Deep Learning The Transformer Architecture The Transformer was introduced in the 2017 paper “Attention Is All You Need.” It replaced recurrent sequence processing with attention-based layers, enabled highly parallel training and became the architectural basis of most modern large language models. Neural Networks & Deep Learning Encoder-Decoder Models Encoder-decoder architectures turn an input into internal representations and then generate an output sequence or structure conditioned on those representations. Neural Networks & Deep Learning Self-Supervised Learning Supervised learning trains a model from examples paired with known labels or target values. Classification and regression are common supervised tasks.

Reference guide and primary sources

Wikipedia is used here as a terminology and history reference guide. Current model versions, institutional statistics and product-specific claims are also linked to first-party or institutional sources because those details can change faster than encyclopedia articles.

← ImageNet and AlexNet Deep Learning →