AI Reference 043 Neural Networks & Deep Learning Wikipedia guided Primary sources linked

Self-Supervised Learning

Definition

Supervised learning trains a model from examples paired with known labels or target values. Classification and regression are common supervised tasks.

What is self-supervised learning?

Self-supervised learning creates training targets from the data itself instead of requiring humans to label every example. The model may predict a missing part, a future part or another transformed view of the same data.

Language models as an example

Autoregressive language models are trained to predict the next token from preceding tokens. The text supplies both the input and the target automatically. Masked-language models instead hide some tokens and train the model to recover them.

Why it enabled foundation models

Human labeling cannot scale to every sentence, image or audio clip on the internet. Self-supervised objectives let models learn from enormous raw datasets, after which smaller amounts of supervised or preference data can adapt the model to specific tasks.

Related terms, defined

Neural Networks & Deep Learning Foundation Models Foundation models are trained on broad data at scale and then adapted or prompted for many downstream tasks. Neural Networks & Deep Learning Multimodal AI Multimodal AI processes or generates multiple data types such as text, images, audio or video in one coordinated system. Neural Networks & Deep Learning Artificial Neural Networks Artificial neural networks are parameterized models built from layers of connected computational units. Training adjusts the parameters to reduce an objective function. Neural Networks & Deep Learning Backpropagation Backpropagation efficiently computes gradients through differentiable neural networks by applying the chain rule from outputs back toward earlier layers. Neural Networks & Deep Learning Convolutional Neural Networks Artificial neural networks are parameterized models built from layers of connected computational units. Training adjusts the parameters to reduce an objective function.

Reference guide and primary sources

Wikipedia is used here as a terminology and history reference guide. Current model versions, institutional statistics and product-specific claims are also linked to first-party or institutional sources because those details can change faster than encyclopedia articles.

← Encoder-Decoder Models Foundation Models →