The Pre-Transformer Era

Word2Vec (Mikolov et al., 2013) learns dense vectors for words such that similar words land near each other in embedding space. The skip-gram variant predicts surrounding context words from a target word using softmax over dot products.

GloVe (Pennington et al., 2014) factorizes a global co-occurrence matrix: it learns vectors whose dot products approximate the log co-occurrence frequency.

Key result: linear algebra on vectors encodes semantic relationships — the famous 'king − man + woman ≈ queen' vector arithmetic.

Critical limitation: these are context-invariant — the word 'bank' gets one fixed vector regardless of whether it means riverbank or financial institution. This was the core flaw that contextual models (ELMo, BERT) later fixed.

Before deep learning, sequence labeling (like finding names in text) relied on statistical graphical models.

Hidden Markov Models (HMMs) model a sequence of hidden states (like parts of speech) that probabilistically emit the observed words. They are generative. Conditional Random Fields (CRFs) model the conditional probability of the state sequence directly given the observations. They are discriminative and handle overlapping features much better than HMMs, becoming the gold standard for NLP until neural networks took over.

Embeddings from Language Models (ELMo, Peters et al.) fixed the fatal flaw of Word2Vec: context-independence.

Instead of a static dictionary lookup, ELMo runs a bidirectional LSTM over the entire sentence. The vector for the word 'bank' is dynamically constructed based on the surrounding words. This was a massive leap forward, proving that pre-training deep language models on unlabeled text creates representations that drastically improve downstream tasks.

An RNN processes a sequence one element at a time, maintaining a hidden state h_t that summarizes everything seen so far. The same weight matrix W_h is reused at every step.

The vanishing gradient problem: during backpropagation, gradients are propagated backward through every time step. Each step multiplies by the Jacobian of the state transition — essentially by W_h repeatedly. If the largest eigenvalue of W_h is less than 1, the product of T such matrices shrinks exponentially toward zero. The network literally cannot 'feel' what happened far back in the sequence.

If the eigenvalue exceeds 1, gradients explode — causing unstable training (addressed by gradient clipping, but the expressiveness limit remains).

LSTMs (Hochreiter & Schmidhuber, 1997) add a cell state and three gates — forget, input, and output — that regulate information flow. Each gate uses a sigmoid function σ(x) = 1/(1+e^{-x}) outputting a value in [0,1]: 0 means 'block everything', 1 means 'pass everything'.

The crucial trick: the cell state is updated additively (C_t = f_t ⊙ C_{t-1} + i_t ⊙ C̃_t). Addition means the gradient of the loss with respect to C_{t-1} flows back as just f_t — not a full matrix multiplication — so gradients travel much further back in time without vanishing.

GRUs (Cho et al., 2014) simplify to two gates (reset and update), offering comparable quality at lower computational cost.

Sequence-to-Sequence (Sutskever et al., 2014) architectures translate an input sequence to an output sequence using two RNNs. The Encoder reads the input and compresses its entire meaning into a single, fixed-size 'context vector'. The Decoder then generates the output from this vector.

The Bottleneck: Forgetting. A single vector cannot hold the details of a 50-word sentence. Performance plummeted on long sequences because the encoder simply couldn't remember the beginning by the time it reached the end.

Seq2Seq models compressed entire input sentences into one fixed-size vector — a severe bottleneck for long inputs. Bahdanau et al. solved this with attention.

At each decoder step: (1) compute an alignment score e_{tj} between the current decoder state s_{t-1} and each encoder hidden state h_j using a small neural network; (2) normalize the scores with softmax to get attention weights α; (3) form a context vector c_t as the weighted sum of encoder states.

The decoder can now dynamically focus on whichever input positions are most relevant for generating each output token — rather than being forced to cram everything into a single vector. This was the conceptual seed from which Transformer self-attention grew.