[Updated on 2026-06-15]
Pre-Transformer NLP was a series of hacks to work around hardware and data constraints. We had good ideas (word analogies, recurrent state, attention) but they hit walls that weren’t architectural. They were compute and memory walls. The Transformer didn’t invent attention. It made attention parallelizable, and that changed everything.
In the beginning, we treated text like a histogram. Bag-of-Words counted term frequencies and called it a day. If “entropy” appeared five times, the document was about physics. TF-IDF (Sparck Jones, 1972) improved this by down-weighting common terms, but the representation was still sparse and semantic-free. “Dog” and “puppy” were as distant as “dog” and “refrigerator.”
Word embeddings (word2vec, Mikolov et al. 2013; GloVe, Pennington et al. 2014) replaced sparse counts with dense vectors, typically 100–300 dimensions. The geometry captured analogies: “King” − “Man” + “Woman” ≈ “Queen.” Training was cheap enough to run on a single CPU overnight. The tradeoff was static vectors: “bank” had one embedding whether the context was a river or a vault. Semantic similarity improved by an order of magnitude. Polysemy was the price.
Recurrent Neural Networks (RNNs) introduced the idea of a hidden state: a vector that updated sequentially as the model read each token. The intuition was that language needs memory. The problem was that training RNNs meant backpropagating through time, and gradients either exploded or vanished over sequences longer than ~20 tokens. LSTMs (Hochreiter & Schmidhuber, 1997) solved this with gated memory cells, and for the next twenty years they were the workhorse of sequence modeling. But they were still sequential: processing token 10 required finishing token 9. A batch of 32 sequences of length 500 took seconds per step even on an 80GB GPU. Throughput was memory-bound, not compute-bound.
ELMo (Peters et al., 2018) was the first sign that contextual representations mattered more than architecture. It stacked biLSTMs on character-level word vectors and raised every benchmark by 3–5 points. The architecture was not new; bidirectional LSTMs had existed for years. What changed was the insight that a single vector per word was insufficient because meaning depends on surrounding context. ELMo computed representations on-the-fly per sentence rather than loading a static lookup table. Every subsequent improvement came from making this contextualization cheaper, deeper, and more parallel.
The Transformer (Vaswani et al., 2017) broke the sequential bottleneck by replacing recurrence with self-attention. Each token attends to every other token in the sequence simultaneously. The computational cost is O(n²·d) per layer (quadratic in sequence length), but the operation is a single large matrix multiplication. That is the exact workload GPUs are built to dominate. We traded the serial memory-bound loop for a parallel compute-bound operation. It was not a smarter algorithm in the cognitive sense. It was a more hardware-aligned one.
BERT (Devlin et al., 2019) applied the Transformer encoder to language understanding with masked language modeling and next-sentence prediction. GPT-2 (Radford et al., 2019) applied the decoder to language generation with the same next-token prediction objective used by language models since Shannon (1948). Three years later, GPT-3 scaled this to 175B parameters — 1,600x the size of BERT-base — and zero-shot performance jumped from narrow benchmarks to broad task transfer. What followed was a decade of empirical work compressed into two years: scaling laws (Kaplan et al., 2020) showed that model performance followed a predictable power law with compute, data, and parameters; Chinchilla (Hoffmann et al., 2022) later refined the optimal ratio to ~20 tokens per parameter, revealing most large models were undertrained by 4x; chain-of-thought reasoning (Wei et al., 2022) showed that intermediate reasoning steps emerge at ~100B scale; instruction tuning (FLAN, Wei et al. 2022) showed that fine-tuning on diverse tasks generalized to unseen tasks; RLHF (Ouyang et al., 2022) aligned generation with human preferences. The architecture that emerged is still the dominant stack as of 2025: a decoder-only Transformer with a next-token objective, predictably scaled, instruction-tuned, and aligned with feedback.
The reason next-token prediction works is not that it compresses text. It works because predicting the next token forces the model to capture the latent structure that generated the text. To predict the next word in a legal argument or a block of Python code, the model needs a representation of the entities, relationships, goals, and constraints in that context. This has been known formally since Shannon (1948): optimal prediction implies optimal compression, and optimal compression implies a model of the source distribution. The cleanest demonstration: Othello-GPT (Li et al., 2023) was trained solely to predict legal moves in Othello with no access to board state. Probing its internal activations revealed an encoded representation of the board — and causal interventions on those activations changed future moves correctly. The model had learned the game’s latent state without ever being told it existed. The engineering surprise (Deletang et al., 2024) is how far this generalizes: language models trained on next-token prediction alone develop representations of spatial reasoning, causal structure, and theory of mind.
We still do not know if next-token prediction alone is sufficient for robust reasoning. Since 2024, a new paradigm has emerged: train models to generate chains of thought via reinforcement learning, then scale inference-time compute at test time (OpenAI o1). Early results suggest reasoning quality improves smoothly with both train-time and test-time compute — a scaling law for thinking, not just for memorizing. It is too early to tell if this is the actual engine of reasoning or another parallelizable hack. What we do know is that the field did not arrive here by pursuing intelligence. It arrived by hitting hardware walls and building the shortest path around them.