What Is a Language Model?

'Large' refers chiefly to the number of parameters (the tunable numbers inside the model — think of them as adjustable knobs) and the volume of training data.

A small language model might have millions of parameters; an LLM has billions to over a trillion. GPT-1 (2018) had 117 million; GPT-3 (2020) had 175 billion — a 1500× jump in two years.

Beyond certain thresholds, qualitatively new behaviors appear ('emergent abilities') — such as solving math problems or writing code that were never explicitly trained. This threshold effect is still debated and not fully understood.

Pre-training: the massive, expensive first phase where the model learns general language patterns from a huge corpus — trillions of tokens. Virtually all the model's 'knowledge' is acquired here. A single pre-training run for a frontier model can cost tens of millions of dollars.

Fine-tuning: a smaller, cheaper later phase that adapts the pre-trained model to specific behaviors — e.g., following instructions (SFT), being helpful and harmless (RLHF/DPO), or answering in a specific domain.

Inference: using the trained model to produce outputs for users. No learning happens; parameters are frozen. The cost is per generated token, paid every time someone uses the model.

• 'LLM = ChatGPT' — ChatGPT is one product built on top of GPT-series LLMs plus fine-tuning and a chat interface. LLMs are a broad class of models from many organizations.

• 'Transformer = LLM' — The Transformer is an architecture; an LLM is a large model trained on language. Vision Transformers exist and are not LLMs. An LLM could in principle use a non-Transformer architecture (Mamba-based models exist).

• 'More parameters = better' — Chinchilla (2022) showed many large models were badly under-trained. A 70B model trained on 1.4T tokens can beat a 175B model trained on only 300B tokens.

• 'Context window = usable context' — RULER (2024) showed models claiming 128K context windows often fail functionally at 32K on multi-hop retrieval tasks.

At their core, neural networks process numbers, not text. A vector is a list of numbers representing a point in high-dimensional space. Words are converted into 'embeddings' — vectors where semantic similarity correlates with geometric proximity.

The dot product measures similarity: it's the projection of one vector onto another. A high positive dot product means the vectors point in similar directions (similar meaning), while near-zero means they are orthogonal (unrelated). This simple operation is the mathematical engine behind attention and embeddings.

How do models learn? Imagine standing on a foggy mountain and wanting to reach the valley (minimum error). Gradient descent means taking a step in the direction of the steepest downward slope.

Backpropagation calculates this slope efficiently. It applies the chain rule of calculus backwards through the network layers to assign 'blame' for the final error to every single parameter, allowing the model to update billions of weights simultaneously.

A single artificial neuron takes multiple inputs, multiplies each by a learned weight, sums them up, and applies a non-linear activation function. It acts as a pattern detector.

A layer is a collection of these neurons acting in parallel. Deep learning stacks these layers: early layers detect simple features (like word frequency), while deeper layers compose them into complex concepts (like sarcasm or logical entailment).

Language models output a probability distribution over the entire vocabulary for the next token. How do we pick one?

Greedy decoding always picks the most likely token, which can lead to repetitive, robotic text. Sampling picks randomly according to the probabilities. Temperature (T) scales these probabilities: T=1 is the original distribution. T < 1 makes the model more confident and deterministic. T > 1 flattens the distribution, increasing randomness and creativity but risking incoherence. Nucleus Sampling (Top-p) restricts the choice to the smallest set of top tokens whose cumulative probability exceeds p, dynamically trimming the 'long tail' of gibberish.

Processing text costs compute. Tokenization is the fast O(N) process of chunking text into IDs using a fixed dictionary. It takes microseconds on a CPU.

Inference is the O(N^2) forward pass through billions of parameters on a GPU. The cost of generating a token is astronomically higher than tokenizing it. This is why LLM pricing is measured in 'per 1M tokens' processed by the model, ignoring the negligible cost of the tokenizer itself.