AI Reference 032 Neural Networks & Deep Learning Wikipedia guided Primary sources linked

Backpropagation

Definition

Backpropagation efficiently computes gradients through differentiable neural networks by applying the chain rule from outputs back toward earlier layers.

What is backpropagation?

Backpropagation is an efficient method for calculating how much each parameter in a differentiable neural network contributed to an output error. It applies the chain rule of calculus from later layers back toward earlier layers.

How it is used in training

A network first performs a forward pass to produce an output and loss. Backpropagation computes gradients of that loss with respect to the parameters. An optimizer such as stochastic gradient descent or Adam then uses those gradients to update the weights.

Historical importance

Versions of reverse-mode differentiation predate modern neural networks, but the 1986 paper by David Rumelhart, Geoffrey Hinton and Ronald Williams helped popularize backpropagation for training multilayer networks. It made it practical for hidden layers to learn useful internal representations.

Related terms, defined

Neural Networks & Deep Learning Convolutional Neural Networks Artificial neural networks are parameterized models built from layers of connected computational units. Training adjusts the parameters to reduce an objective function. Neural Networks & Deep Learning Recurrent Neural Networks Artificial neural networks are parameterized models built from layers of connected computational units. Training adjusts the parameters to reduce an objective function. Neural Networks & Deep Learning Long Short-Term Memory Networks LSTM networks are recurrent neural networks with gated memory mechanisms designed to preserve and control information across longer sequences. Neural Networks & Deep Learning Word Embeddings Word embeddings represent words or tokens as dense numerical vectors whose geometry captures statistical relationships learned from data. Neural Networks & Deep Learning ImageNet and AlexNet AlexNet's strong result in the 2012 ImageNet competition helped demonstrate the effectiveness of deep convolutional neural networks trained on GPUs and large image datasets.

Reference guide and primary sources

Wikipedia is used here as a terminology and history reference guide. Current model versions, institutional statistics and product-specific claims are also linked to first-party or institutional sources because those details can change faster than encyclopedia articles.

← Artificial Neural Networks Convolutional Neural Networks →