AI Reference 036 Neural Networks & Deep Learning Wikipedia guided Primary sources linked

Word Embeddings

Definition

Word embeddings represent words or tokens as dense numerical vectors whose geometry captures statistical relationships learned from data.

What is a word embedding?

A word embedding is a dense numerical vector representing a word or token. Instead of assigning each word an unrelated symbolic ID, an embedding places words in a continuous vector space learned from usage patterns.

What the geometry can capture

Words used in similar contexts tend to acquire similar vectors. Methods such as Word2Vec, introduced by Tomas Mikolov and colleagues in 2013, made vector relationships famous because semantic and syntactic patterns could appear as directions in the space.

How embeddings evolved

Early systems often assigned one static vector per word. Transformer models create contextual embeddings, so the representation of a token can change depending on the surrounding text. Separate embedding models are also widely used for search and retrieval.

Related terms, defined

Neural Networks & Deep Learning ImageNet and AlexNet AlexNet's strong result in the 2012 ImageNet competition helped demonstrate the effectiveness of deep convolutional neural networks trained on GPUs and large image datasets. Neural Networks & Deep Learning GPUs and the Rise of Deep Learning GPUs perform many numerical operations in parallel, which fits the matrix and tensor workloads used in neural-network training and inference. Neural Networks & Deep Learning Deep Learning Deep learning uses neural networks with multiple layers of learned representations. Its modern rise combined larger datasets, better training methods and parallel hardware. Neural Networks & Deep Learning Attention Mechanisms Attention computes context-dependent weighted combinations of representations, allowing a model to emphasize information relevant to the current prediction. Neural Networks & Deep Learning The Transformer Architecture The Transformer was introduced in the 2017 paper “Attention Is All You Need.” It replaced recurrent sequence processing with attention-based layers, enabled highly parallel training and became the architectural basis of most modern large language models.

Reference guide and primary sources

Wikipedia is used here as a terminology and history reference guide. Current model versions, institutional statistics and product-specific claims are also linked to first-party or institutional sources because those details can change faster than encyclopedia articles.

← Long Short-Term Memory Networks ImageNet and AlexNet →