AI Reference 034 Neural Networks & Deep Learning Wikipedia guided Primary sources linked

Recurrent Neural Networks

Definition

Artificial neural networks are parameterized models built from layers of connected computational units. Training adjusts the parameters to reduce an objective function.

What is a recurrent neural network?

A recurrent neural network, or RNN, processes a sequence while carrying a hidden state from one step to the next. The state acts as a numerical summary of previous inputs.

Why recurrence was useful

Language, speech and time series have order. RNNs provided a natural way to process one element at a time while allowing earlier information to influence later outputs. They were widely used before transformers became dominant in large-scale language modeling.

The long-range problem

Basic RNNs can suffer from vanishing or exploding gradients, making long-distance dependencies difficult to learn. LSTM and GRU architectures added gates that improved long-term information flow. Transformers later removed recurrence entirely for many sequence tasks.

Related terms, defined

Neural Networks & Deep Learning Long Short-Term Memory Networks LSTM networks are recurrent neural networks with gated memory mechanisms designed to preserve and control information across longer sequences. Neural Networks & Deep Learning Word Embeddings Word embeddings represent words or tokens as dense numerical vectors whose geometry captures statistical relationships learned from data. Neural Networks & Deep Learning ImageNet and AlexNet AlexNet's strong result in the 2012 ImageNet competition helped demonstrate the effectiveness of deep convolutional neural networks trained on GPUs and large image datasets. Neural Networks & Deep Learning GPUs and the Rise of Deep Learning GPUs perform many numerical operations in parallel, which fits the matrix and tensor workloads used in neural-network training and inference. Neural Networks & Deep Learning Deep Learning Deep learning uses neural networks with multiple layers of learned representations. Its modern rise combined larger datasets, better training methods and parallel hardware.

Reference guide and primary sources

Wikipedia is used here as a terminology and history reference guide. Current model versions, institutional statistics and product-specific claims are also linked to first-party or institutional sources because those details can change faster than encyclopedia articles.

← Convolutional Neural Networks Long Short-Term Memory Networks →