GPT: Generative Pre-Training

Chapter Overview

GPT (Generative Pre-trained Transformer) pioneered decoder-only transformer architectures for autoregressive language modeling. This chapter traces the evolution from GPT-1 through GPT-4, covering architecture, pre-training, scaling, few-shot learning, and emergent abilities.

Learning Objectives

1. Understand GPT's decoder-only architecture
2. Implement autoregressive language modeling
3. Apply in-context learning and few-shot prompting
4. Analyze scaling laws and emergent abilities
5. Compare GPT variants (GPT-1, GPT-2, GPT-3, GPT-4)
6. Understand instruction tuning and RLHF

GPT Architecture

Decoder-Only Transformers

The GPT architecture represents a fundamental departure from the encoder-decoder paradigm that dominated sequence-to-sequence models. Rather than using separate encoder and decoder stacks, GPT employs only transformer decoder blocks, creating a purely autoregressive language model. This architectural choice has profound implications for both the model's capabilities and its computational characteristics.

The core innovation lies in the attention mechanism's masking pattern. GPT uses causal masking, which prevents each position from attending to future positions in the sequence. Mathematically, when computing attention scores $\mS = \mQ \mK\transpose$, a mask is applied such that $S_{ij} = -\infty$ for all $j > i$. After the softmax operation, these masked positions have zero attention weight, ensuring that the representation at position $i$ depends only on tokens at positions $1$ through $i$. This causal constraint is essential for autoregressive generation, where the model must predict the next token without access to future context.

Unlike the original transformer architecture which included cross-attention layers to attend from decoder to encoder, GPT eliminates cross-attention entirely. Each decoder block contains only a masked self-attention layer followed by a position-wise feed-forward network. This simplification reduces architectural complexity while maintaining the transformer's parallel processing advantages. The self-attention layer allows each position to gather information from all previous positions simultaneously, avoiding the sequential bottleneck of recurrent networks.

GPT-2 and later versions introduced an important architectural refinement: pre-normalization. Rather than applying layer normalization after each sub-layer (post-norm), pre-norm applies normalization before the attention and feed-forward operations. This seemingly minor change significantly improves training stability for deep networks. In the pre-norm configuration, the residual path carries the original signal without normalization, providing a clean gradient path during backpropagation. This enables training of much deeper models without the gradient instability that plagued earlier architectures.

The distinction between GPT and BERT architectures illuminates different modeling philosophies. BERT employs bidirectional attention, allowing each position to attend to the entire sequence including future tokens. This bidirectionality enables rich contextual representations ideal for understanding tasks like classification and question answering. However, bidirectional attention is incompatible with autoregressive generation—the model cannot predict the next token if it has already seen it. GPT's unidirectional causal attention sacrifices bidirectional context but gains the ability to generate coherent text autoregressively. This trade-off reflects the fundamental tension between understanding (BERT) and generation (GPT) in language modeling.

GPT Model Sizes

The evolution of GPT models demonstrates the remarkable scaling properties of transformer architectures. Each generation increased model capacity by orders of magnitude, revealing new capabilities that emerged only at larger scales. Understanding the progression from GPT-1 through GPT-3 provides insight into the relationship between model size and performance.

GPT-1, introduced in 2018, established the decoder-only pre-training paradigm with 117 million parameters. The architecture used 12 transformer layers with hidden dimension $d = 768$ and 12 attention heads, processing sequences up to 512 tokens. While modest by today's standards, GPT-1 demonstrated that unsupervised pre-training on large text corpora followed by task-specific fine-tuning could achieve strong performance across diverse NLP tasks. The model was trained on BookCorpus, a dataset of approximately 7,000 unpublished books containing 800 million words. This training data, while substantial for 2018, would be considered quite limited compared to later models.

GPT-2, released in 2019, expanded the scaling experiment by training four model sizes ranging from 117 million to 1.5 billion parameters. The smallest GPT-2 matched GPT-1's architecture, while GPT-2 XL scaled to 48 layers with hidden dimension $d = 1600$ and 25 attention heads. The context window doubled to 1024 tokens, enabling the model to maintain coherence over longer passages. More significantly, GPT-2 was trained on WebText, a dataset of 40 GB containing 8 million web pages. This diverse training data, scraped from outbound links on Reddit with at least 3 karma, provided much broader coverage of topics and writing styles than BookCorpus. GPT-2's key finding was that larger models trained on more diverse data could perform many tasks zero-shot, without any task-specific fine-tuning—a surprising emergent capability.

GPT-3, unveiled in 2020, represented a massive leap to 175 billion parameters. The architecture scaled to 96 layers with hidden dimension $d = 12288$ and 96 attention heads, processing sequences of 2048 tokens. The parameter count increased by more than 100× compared to GPT-2 XL, requiring fundamentally different training infrastructure. GPT-3 was trained on approximately 300 billion tokens drawn from Common Crawl (filtered), WebText2, Books1, Books2, and Wikipedia, totaling roughly 570 GB of text. The training used a single pass through this massive dataset rather than multiple epochs, reflecting the compute-optimal insight that data diversity matters more than repeated exposure to the same examples. GPT-3's most striking capability was few-shot learning: the model could perform new tasks by conditioning on a few examples in the prompt, without any parameter updates. This in-context learning ability scaled dramatically with model size, with GPT-3 175B far outperforming smaller variants.

GPT-4, released in 2023, marked another architectural evolution, though OpenAI disclosed fewer details. Estimates suggest the model uses a mixture-of-experts architecture with 1 to 1.7 trillion total parameters, though only a fraction are active for any given input. The context window expanded dramatically to 8,192 tokens in the standard version and 32,768 tokens in the extended version, enabling the model to process entire documents or codebases. GPT-4 demonstrated significant improvements in reasoning, factual accuracy, and instruction following, suggesting that architectural innovations beyond pure parameter scaling contributed to its capabilities.

GPT-1 (2018):

• Layers: $L = 12$, Hidden: $d = 768$, Heads: $h = 12$
• Parameters: 117M
• Context: 512 tokens

GPT-2 (2019):

• Small: 117M, Medium: 345M, Large: 762M, XL: 1.5B
• GPT-2 XL: $L=48$, $d=1600$, $h=25$
• Context: 1024 tokens

GPT-3 (2020):

• Small: 125M to XL: 175B
• GPT-3 175B: $L=96$, $d=12288$, $h=96$
• Context: 2048 tokens
• Parameters: 175 billion!

GPT-4 (2023):

• Architecture details not fully disclosed
• Estimated: 1-1.7 trillion parameters (mixture of experts)
• Context: 8K (standard), 32K (extended)

Pre-Training: Autoregressive Language Modeling

Training Objective

Autoregressive language modeling forms the foundation of GPT's pre-training approach. Unlike masked language modeling used in BERT, which predicts randomly masked tokens using bidirectional context, autoregressive modeling predicts each token based solely on preceding tokens. This objective aligns naturally with text generation tasks and enables the model to learn the statistical structure of language through next-token prediction.

The training objective maximizes the likelihood of each token given all previous tokens in the sequence. For a sequence $\vx = [x_1, x_2, \ldots, x_n]$, the model learns to maximize the joint probability $P(x_1, x_2, \ldots, x_n) = \prod_{i=1}^{n} P(x_i | x_1, \ldots, x_{i-1})$. Taking the logarithm converts this product into a sum, yielding the standard language modeling loss. This formulation has an elegant interpretation: the model learns to compress the training data by assigning high probability to observed sequences, with the negative log-likelihood measuring the number of bits required to encode the data under the model's distribution.

The implementation leverages the transformer's parallel processing capabilities through teacher forcing. Rather than generating tokens sequentially during training, the entire sequence is processed in a single forward pass. The input sequence $[x_1, x_2, \ldots, x_n]$ is fed to the model, which produces hidden representations for all positions simultaneously. The causal attention mask ensures that position $i$ cannot attend to positions $j > i$, maintaining the autoregressive property despite parallel computation. The model's output at position $i$ is trained to predict token $x_{i+1}$, creating $n-1$ training signals from a single sequence of length $n$. This parallel training is dramatically more efficient than sequential generation, enabling large-scale pre-training on massive text corpora.

The cross-entropy loss is computed at each position by comparing the model's predicted distribution over the vocabulary with the true next token. For position $i$ with hidden state $\vh_i$, the model computes logits $\vz_i = \vh_i \mW_{\text{out}}$ where $\mW_{\text{out}} \in \R^{d_{\text{model}} \times V}$ projects to vocabulary size $V$. Applying softmax yields a probability distribution $P(x_{i+1} | x_1, \ldots, x_i) = \text{softmax}(\vz_i)$. The loss for this position is $-\log P(x_{i+1} | x_1, \ldots, x_i)$, and the total loss sums over all positions. This formulation naturally handles variable-length sequences and provides dense training signal from every token in the corpus.

Implementation:

1. Input: $[x_1, x_2, \ldots, x_n]$
2. Target: $[x_2, x_3, \ldots, x_{n+1}]$ (shifted by 1)
3. Causal mask: Position $i$ cannot attend to $j > i$
4. Cross-entropy loss at each position

Pre-Training Data

The scale and diversity of pre-training data have proven critical to GPT's capabilities. Each generation of GPT models trained on progressively larger and more diverse text corpora, revealing that data quality and quantity both matter significantly for downstream performance.

GPT-1 was trained on BooksCorpus, a collection of approximately 7,000 unpublished books from various genres including adventure, fantasy, and romance. This dataset contained roughly 800 million words, providing coherent long-form text that helped the model learn narrative structure and long-range dependencies. The choice of books as training data reflected the hypothesis that long-form text with coherent structure would be more valuable than shorter, disconnected documents. However, the relatively narrow domain coverage limited the model's exposure to diverse topics and writing styles.

GPT-2 marked a significant shift in data philosophy with the creation of WebText, a dataset of 40 GB containing text from 8 million web pages. The data was collected by scraping outbound links from Reddit posts with at least 3 karma, using social curation as a quality filter. This approach yielded much more diverse content spanning news articles, tutorials, discussions, and creative writing across virtually all topics. The 40 GB corpus represented approximately 10 billion tokens, more than an order of magnitude larger than BooksCorpus. This scale and diversity enabled GPT-2 to demonstrate surprising zero-shot capabilities on tasks it had never been explicitly trained to perform.

GPT-3 scaled data collection to unprecedented levels, training on approximately 300 billion tokens drawn from multiple sources. The training mixture included Common Crawl (filtered to remove low-quality content), WebText2 (an expanded version of GPT-2's dataset), Books1, Books2, and Wikipedia. The total dataset size reached roughly 570 GB of text. Critically, GPT-3 was trained for a single epoch over this massive dataset rather than multiple passes over smaller data. This decision reflected emerging understanding of scaling laws: given fixed compute budget, it is often better to train on more diverse data once than to repeatedly train on the same limited data. The single-epoch approach also reduced the risk of memorizing specific training examples, though concerns about data contamination and memorization remained.

The composition of GPT-3's training data was carefully weighted, with higher-quality sources sampled more frequently. Common Crawl, despite being the largest source, was downweighted due to quality concerns, while Wikipedia and books received higher sampling rates. This weighting scheme balanced scale with quality, ensuring the model learned from both broad web text and curated high-quality sources. The exact mixing ratios and filtering procedures significantly impacted model performance, though these details were not fully disclosed.

GPT-1: BooksCorpus (7,000 books, $\approx$ 800M words)

GPT-2: WebText (40GB, 8M web pages)

GPT-3: Common Crawl (filtered), WebText2, Books1, Books2, Wikipedia

• Total: $\approx$ 570GB text
• Tokens: $\approx$ 300 billion
• Training: Single pass (not multiple epochs)

Training Infrastructure and Costs

Training GPT models at scale requires massive infrastructure, with costs increasing by orders of magnitude between model generations. GPT-2's training on 32 TPU v3 chips for one week cost approximately \$50,000---accessible to well-funded labs. GPT-3's training on 10,000+ V100 GPUs for one month cost an estimated \$4--12 million, consuming 1,287~MWh of energy and requiring sophisticated distributed training strategies (model, pipeline, and data parallelism). For a detailed treatment of distributed training infrastructure and cost estimation, see Chapter~[ref] (Training Transformers).

GPT-2 Training:

• Hardware: 32 TPU v3 chips ($\approx$ 13.4 PFLOPS)
• Training time: $\approx$ 1 week
• Cost: $\approx$ \$50,000
• Batch size: 512 sequences $\times$ 1024 tokens
• Learning rate: Cosine decay with warmup

GPT-3 Training:

• Hardware: 10,000+ V100 GPUs (estimated, $>$ 1 exaFLOP)
• Training time: $\approx$ 1 month
• Cost: \$4-12 million (estimated)
• Energy consumption: 1,287 MWh
• Requires model parallelism, pipeline parallelism, and data parallelism
• High-bandwidth interconnects (NVLink, InfiniBand) essential

In-Context Learning and Few-Shot Prompting

Autoregressive Generation with KV Caching

Before exploring in-context learning, we must understand how GPT generates text autoregressively. The generation process differs fundamentally from training, as tokens are produced sequentially rather than in parallel. Naive implementation of autoregressive generation is extremely inefficient, but key-value caching provides dramatic speedups that make interactive generation practical.

During generation, the model produces one token at a time. Starting with a prompt, the model computes attention over all prompt tokens to generate the first new token. Then it appends this token to the sequence and computes attention over all tokens (prompt plus generated) to produce the second token. This continues until reaching a stopping condition like a maximum length or end-of-sequence token. The critical inefficiency is that each generation step recomputes attention for all previous tokens, even though their key and value representations never change.

Consider generating a sequence of length $T$ tokens. The first step processes $n_0$ prompt tokens, computing keys and values for all positions. The second step processes $n_0 + 1$ tokens, recomputing the same keys and values for the prompt plus computing them for the new token. By step $T$, we have computed keys and values for the prompt tokens $T$ times, despite them being identical each time. The total computation grows quadratically: $\sum_{t=1}^{T} (n_0 + t) = Tn_0 + T(T+1)/2 \approx Tn_0 + T^2/2$ forward passes through the attention mechanism.

Key-value caching eliminates this redundancy by storing the computed keys and values for all previous tokens. When generating token $t$, we only compute keys and values for the new token at position $t$, then concatenate with the cached keys and values from positions $1$ through $t-1$. The attention computation at position $t$ uses the full key and value matrices, but we avoid recomputing the cached portions. This reduces the computation from quadratic to linear in the generation length.

The memory requirements for KV caching scale with the sequence length, number of layers, and model dimension. For each layer, we must store key and value matrices of shape $[n_{\text{current}}, d_{\text{model}}]$ where $n_{\text{current}}$ is the current sequence length. With $L$ layers and hidden dimension $d$, the cache requires $2 \times L \times n_{\text{current}} \times d$ values. For GPT-2 with 12 layers, dimension 768, and sequence length 1024, the cache occupies $2 \times 12 \times 1024 \times 768 = 18{,}874{,}368$ values, or approximately 75 MB in FP32 per sequence. This is modest compared to model parameters (440 MB for GPT-2), but grows linearly with batch size and sequence length.

The generation speed improvement from KV caching is dramatic. Without caching, generating $T$ tokens requires $O(T^2)$ operations. With caching, it requires $O(T)$ operations. For GPT-2 generating 100 tokens, this represents a 50× speedup in theory. In practice, the speedup is somewhat less due to memory bandwidth limitations and the overhead of managing the cache, but 10-20× speedups are typical. This transforms generation from painfully slow (1-2 tokens per second) to interactive (20-50 tokens per second) on modern GPUs.

Batch generation introduces additional trade-offs. Processing multiple sequences in parallel amortizes the cost of loading model parameters and improves GPU utilization. However, the KV cache memory scales linearly with batch size. For GPT-2 with batch size 32 and sequence length 1024, the cache requires $32 \times 75\text{ MB} = 2.4\text{ GB}$. Combined with model parameters and activations, this can exhaust GPU memory. Practitioners must balance batch size against sequence length and model size to fit within memory constraints. Dynamic batching, where sequences of different lengths are grouped together, can improve efficiency by allowing longer sequences when the batch is small and more sequences when they are short.

Generation algorithm with KV caching:

1. Process prompt tokens $[x_1, \ldots, x_{n_0}]$ in parallel, computing and caching keys/values for all layers
2. For generation step $t = 1, 2, \ldots, T$:
   1. Compute keys/values only for new token at position $n_0 + t$
   2. Concatenate with cached keys/values from positions $1$ to $n_0 + t - 1$
   3. Compute attention using full key/value matrices
   4. Generate next token from output distribution
   5. Append new keys/values to cache
3. Return generated sequence $[x_{n_0+1}, \ldots, x_{n_0+T}]$

Memory requirements for KV cache:

For GPT-2 (12 layers, 768 dim, 1024 tokens, batch 1, FP32):

For GPT-3 (96 layers, 12288 dim, 2048 tokens, batch 1, FP16):

Generation speed comparison:

• Without caching: $\sim$1-2 tokens/sec (recomputes all previous tokens)
• With caching: $\sim$20-50 tokens/sec for GPT-2 on V100
• With caching: $\sim$10-15 tokens/sec for GPT-3 on A100 (batch 1)
• Batch generation: Higher throughput (tokens/sec) but same latency per sequence

Zero-Shot, One-Shot, Few-Shot

Zero-Shot, One-Shot, Few-Shot

GPT-3's most remarkable capability is in-context learning: the ability to perform new tasks by conditioning on examples provided in the prompt, without any parameter updates or gradient descent. This emergent behavior was not explicitly trained for, yet it scales dramatically with model size, suggesting that large language models develop meta-learning capabilities through pre-training alone.

Zero-shot learning provides only a task description without examples. The model must infer the desired behavior from the natural language instruction alone. For translation, a zero-shot prompt might simply state "Translate English to French:" followed by the source text. The model must recognize the task from the instruction and generate an appropriate translation. Zero-shot performance varies widely across tasks—GPT-3 performs well on common tasks like translation and summarization but struggles with specialized or ambiguous tasks where the instruction alone provides insufficient specification.

Translate English to French:
sea otter =>

One-shot learning adds a single example demonstrating the desired input-output mapping. This single example often dramatically improves performance by clarifying the task format, output style, and level of detail expected. For translation, showing one English-French pair helps the model understand not just that translation is required, but also the desired formality level, whether to include punctuation, and how to handle proper nouns. The improvement from zero-shot to one-shot is often larger than from one-shot to few-shot, suggesting that the first example resolves most of the task ambiguity.

Translate English to French:
sea otter => loutre de mer
cheese =>

Few-shot learning provides multiple examples, typically between 10 and 100 depending on the task complexity and context window size. Additional examples help the model learn task-specific patterns, edge cases, and output formatting. For classification tasks, few-shot examples should cover all classes to avoid bias toward classes seen more frequently. For generation tasks, examples demonstrate the desired output length, style, and structure. The performance improvement from few-shot learning scales with both the number of examples and the model size—larger models extract more information from the same examples.

Translate English to French:
sea otter => loutre de mer
peppermint => menthe poivrée
plush giraffe => girafe en peluche
cheese =>

The mechanism underlying in-context learning remains partially mysterious. The model is not performing gradient descent or updating parameters—it processes the prompt in a single forward pass. Instead, the model appears to perform a form of implicit Bayesian inference, using the examples to narrow down the space of possible tasks and then applying the inferred task to the query. The attention mechanism plays a crucial role, allowing later tokens to attend to earlier examples and extract relevant patterns. Larger models have more capacity to represent complex task distributions and perform more sophisticated inference, explaining why few-shot learning improves dramatically with scale.

The practical implications are profound. In-context learning enables rapid adaptation to new tasks without fine-tuning, which requires labeled data, computational resources, and time. Users can deploy GPT-3 on novel tasks by simply crafting appropriate prompts with examples. This has spawned the field of prompt engineering, where practitioners carefully design prompts to elicit desired behaviors. However, in-context learning has limitations—it cannot match fine-tuned performance on tasks with abundant training data, and it is sensitive to example selection and ordering. The examples must fit within the context window, limiting the amount of task-specific information that can be provided.

Emergent Abilities

As language models scale to billions and hundreds of billions of parameters, they exhibit emergent abilities—capabilities that appear suddenly at certain scale thresholds rather than improving gradually. These emergent behaviors were not explicitly programmed or trained for, yet they arise naturally from the combination of scale, architecture, and training data. Understanding emergence is crucial for predicting what capabilities future models might develop and for identifying the minimum scale required for specific applications.

Few-shot learning itself is an emergent ability. Models with fewer than 1 billion parameters show minimal few-shot learning capability—providing examples in the prompt barely improves performance over zero-shot. Between 1 billion and 10 billion parameters, few-shot learning begins to emerge, with clear improvements from adding examples. By 100 billion parameters, few-shot learning becomes highly effective, with GPT-3 175B demonstrating strong performance on many tasks with just 10-20 examples. This non-linear scaling suggests a phase transition in the model's internal representations, where sufficient capacity enables a qualitatively different form of processing.

Chain-of-thought reasoning represents another striking emergent ability. When prompted to show its reasoning step-by-step before providing an answer, models around 100 billion parameters begin to solve complex multi-step problems that smaller models cannot. For arithmetic word problems, asking the model to "think step by step" dramatically improves accuracy. The model generates intermediate reasoning steps, then uses those steps to arrive at the final answer. This capability appears suddenly—models below a certain scale show no benefit from chain-of-thought prompting, while larger models show substantial improvements. The emergence suggests that large models develop internal mechanisms for decomposing complex problems into simpler sub-problems.

Complex instruction following emerges only in the largest models. GPT-3 175B can follow multi-part instructions, maintain consistency across long generations, and adapt its behavior based on nuanced prompt details. Smaller models often ignore parts of complex instructions or fail to maintain consistency. This capability is essential for practical applications where users need fine-grained control over model behavior. The emergence of instruction following motivated the development of instruction-tuned models like InstructGPT, which further enhance this capability through supervised fine-tuning and reinforcement learning.

The scaling curve for most capabilities follows a smooth power law—performance improves predictably as model size increases. However, emergent abilities show sharp phase transitions where performance jumps discontinuously at certain scales. This creates challenges for predicting model capabilities: extrapolating from smaller models may underestimate the capabilities of larger models. It also raises questions about what other abilities might emerge at even larger scales. Some researchers hypothesize that abilities like true reasoning, planning, and causal understanding might emerge at scales beyond current models, while others argue that architectural changes or different training objectives are necessary.

The mechanism underlying emergence remains debated. One hypothesis is that emergent abilities require a minimum representational capacity—below this threshold, the model cannot represent the necessary abstractions, while above it, the ability appears. Another hypothesis focuses on the training dynamics: certain capabilities require seeing specific patterns in the training data a minimum number of times, which only occurs when training on massive datasets. A third perspective suggests that emergence is partially an artifact of evaluation metrics—capabilities may improve gradually, but threshold-based metrics (like exact match accuracy) show discontinuous jumps.

Abilities that appear suddenly at certain scales:

• Few-shot learning: Emerges around 1B-10B parameters
• Chain-of-thought reasoning: Emerges around 100B parameters
• Complex instruction following: Largest models

Scaling curve: Performance on many tasks follows smooth power law, but some tasks show sharp phase transitions.

Scaling Laws

Parameter Scaling

The relationship between model size and performance follows remarkably predictable patterns, enabling researchers to forecast the capabilities of larger models before building them. These scaling laws have become central to modern AI research, guiding decisions about how to allocate computational resources between model size, training data, and training time.

The fundamental scaling law relates model performance, measured by loss on held-out data, to the number of parameters. Empirically, the loss follows a power law:

Performance (measured by loss) scales as:

The practical implications are profound. The power law allows researchers to predict the performance of a 1 trillion parameter model by extrapolating from experiments with 1 billion and 10 billion parameter models. This predictability has motivated continued scaling efforts, as the returns to scale remain consistent even at enormous sizes. However, the exponent $\alpha \approx 0.076$ means that improvements slow as models grow—achieving the same loss reduction requires exponentially more parameters. Reducing loss by half requires increasing parameters by a factor of $(2)^{1/0.076} \approx 150$, making continued progress increasingly expensive.

The scaling law applies specifically to the pre-training loss, which measures how well the model predicts the next token. Downstream task performance does not always scale as smoothly—some tasks show rapid improvement with scale while others plateau. This discrepancy arises because pre-training loss captures general language understanding, while specific tasks may require capabilities that emerge only at certain scales or that are not well-measured by next-token prediction. Nevertheless, pre-training loss remains the most reliable predictor of overall model capability.

Importantly, the scaling law holds only when other factors are not bottlenecks. If the training data is too small, the model will overfit and the scaling law breaks down. If the training time is too short, the model will not converge and performance will be suboptimal. The scaling laws assume that data and compute are scaled appropriately with model size, a condition that is not always met in practice.

Implications:

• Every 10× increase in parameters $\to$ consistent loss reduction
• No sign of saturation up to 175B parameters
• Motivates continued scaling

Compute-Optimal Training

While the parameter scaling law shows that larger models achieve better performance, it does not address the question of how to optimally allocate a fixed compute budget. Should we train a very large model on limited data, or a smaller model on more data? The Chinchilla paper provided a surprising answer that has reshaped thinking about model scaling.

The Chinchilla findings, based on training over 400 language models ranging from 70 million to 16 billion parameters, revealed that for a given compute budget $C$, the optimal allocation scales both model size and training data:

This square-root scaling means that if you increase compute by 100×, you should increase both model size and training data by 10×. Critically, this implies that model size and data should scale equally—doubling compute should double both parameters and training tokens.

Chinchilla findings: For compute budget $C$, optimal allocation is:

Applying this formula to GPT-3 reveals a striking conclusion: the model was significantly over-parameterized for its training compute. GPT-3 used 175 billion parameters trained on 300 billion tokens. According to Chinchilla scaling laws, the same compute budget would be better spent on an 80 billion parameter model trained on 1.4 trillion tokens. This smaller, better-trained model would achieve lower loss and better downstream performance than GPT-3, despite having less than half the parameters.

This finding explains why many large models are over-parameterized and under-trained. The focus on parameter count as a headline metric incentivized building the largest possible models, even when training data was insufficient. The Chinchilla results suggest that future models should prioritize data quality and quantity alongside parameter scaling. This has motivated efforts to curate larger, higher-quality training datasets and to train models for more steps on existing data.

The compute-optimal scaling also has implications for inference costs. Larger models are more expensive to serve, requiring more memory and compute per token generated. If a smaller, better-trained model achieves the same performance, it will be cheaper to deploy. This economic consideration is increasingly important as language models move from research to production applications serving millions of users.

However, the Chinchilla findings come with caveats. The optimal allocation depends on the relative costs of training versus inference. If inference costs dominate (as in production systems serving many users), a larger model trained on less data may be preferable because it achieves better performance per inference FLOP. The optimal allocation also depends on the availability of high-quality training data—if data is limited or expensive to collect, training a larger model on available data may be the only option.

GPT-3 analysis:

• 175B parameters trained on 300B tokens
• Chinchilla suggests: 80B parameters on 1.4T tokens would be better
• Many large models are over-parameterized, under-trained

The future direction suggested by these findings is clear: smaller models trained on more data. This approach reduces training costs (fewer parameters to update), reduces inference costs (smaller models to serve), and improves performance (better training efficiency). The challenge lies in collecting and curating the massive datasets required—1.4 trillion tokens is nearly 5× the data used for GPT-3, requiring extensive web scraping, filtering, and deduplication. Nevertheless, the Chinchilla findings have fundamentally shifted the scaling paradigm from "bigger is better" to "balanced scaling is optimal."

Hardware Requirements for Inference

While training requirements determine whether a model can be built, inference requirements determine whether it can be deployed. Understanding the hardware needed to serve GPT models is essential for practitioners considering which models to use in production and for researchers designing new architectures.

GPT-2 with 1.5 billion parameters represents the upper end of models that can be served efficiently on consumer hardware. In FP16 precision, the model parameters occupy $1.5 \times 10^9 \times 2 = 3$ GB of memory. Adding the KV cache for a sequence of 1024 tokens requires approximately 75 MB per sequence, and activations for a single forward pass add another 100-200 MB. A single NVIDIA V100 GPU with 16 GB of memory can comfortably serve GPT-2 with batch sizes of 4-8 sequences, achieving generation speeds of approximately 50 tokens per second per sequence. This makes GPT-2 practical for real-time applications like chatbots, code completion, and interactive writing assistants.

The generation speed of 50 tokens per second on a V100 reflects several factors. The V100 provides 125 TFLOPS of FP16 performance, but actual utilization is typically 30-50\% for autoregressive generation due to the sequential nature of the computation and memory bandwidth limitations. Each token generation requires a forward pass through all 48 layers, computing attention over the growing sequence length. With KV caching, the computation per token is roughly constant, but memory bandwidth for loading the cache and model parameters becomes the bottleneck. Batch processing multiple sequences in parallel improves throughput by amortizing parameter loading, but latency per sequence remains constant.

GPT-2 (1.5B) Inference:

• Memory (FP16): 3 GB parameters + 75 MB KV cache per sequence + 200 MB activations
• Hardware: Single V100 (16 GB) or RTX 3090 (24 GB)
• Batch size: 4-8 sequences on V100
• Generation speed: $\sim$50 tokens/sec per sequence
• Latency: $\sim$20 ms per token
• Practical for: Real-time applications, edge deployment

GPT-3 with 175 billion parameters presents dramatically different challenges. In FP16 precision, the parameters alone require $175 \times 10^9 \times 2 = 350$ GB of memory. No single GPU can hold the entire model—even the largest NVIDIA A100 with 80 GB falls far short. Model parallelism is essential, splitting the model across multiple GPUs. A minimum of 8× A100 (80 GB) GPUs is required just to hold the parameters, with each GPU storing approximately 44 GB of model weights. The KV cache for GPT-3 with 2048 tokens requires approximately 9.7 GB per sequence, further constraining batch sizes. With 8 GPUs, the total available memory is 640 GB, leaving roughly 290 GB for KV cache and activations after storing parameters—enough for batch sizes of 20-30 sequences.

The generation speed for GPT-3 is significantly slower than GPT-2, despite using more powerful hardware. With batch size 1 on 8× A100 GPUs, GPT-3 generates approximately 10 tokens per second. The slowdown reflects several factors. First, the model is 100× larger, requiring 100× more computation per token. Second, model parallelism introduces communication overhead—activations must be transferred between GPUs at each layer, consuming bandwidth and adding latency. Third, the larger KV cache requires more memory bandwidth to load at each generation step. Increasing batch size improves throughput (total tokens per second across all sequences) but does not reduce latency per sequence.

GPT-3 (175B) Inference:

• Memory (FP16): 350 GB parameters + 9.7 GB KV cache per sequence
• Hardware: Minimum 8× A100 (80 GB), often 16× for production
• Model parallelism: Required—split across GPUs
• Batch size: 1-4 sequences per 8-GPU node (memory constrained)
• Generation speed: $\sim$10 tokens/sec per sequence (batch 1)
• Latency: $\sim$100 ms per token
• Cost: \$0.02-0.06 per 1000 tokens (cloud pricing)

The high cost of GPT-3 inference has motivated extensive optimization efforts. Quantization to INT8 or INT4 reduces memory requirements by 2-4×, enabling larger batch sizes or smaller hardware configurations. However, quantization requires careful calibration to avoid accuracy degradation, and not all operations benefit equally—attention computations are particularly sensitive to reduced precision. Distillation, where a smaller model is trained to mimic GPT-3's outputs, can achieve 90-95\% of the performance with 10× fewer parameters, dramatically reducing inference costs. Sparse models, where only a subset of parameters are active for each input, offer another path to efficiency.

The economics of serving GPT-3 at scale are daunting. A single 8× A100 node costs approximately \$30,000-50,000 to purchase or \$20-30 per hour to rent from cloud providers. At 10 tokens per second, a single node can serve roughly 36,000 tokens per hour, or 864,000 tokens per day. For applications serving millions of users, dozens or hundreds of nodes are required, with costs reaching millions of dollars per month. This has created a market for inference-optimized models and specialized hardware, as well as prompting research into more efficient architectures that maintain capability while reducing computational requirements.

Why GPT-3 inference is expensive:

• Memory: 350 GB parameters require multiple high-end GPUs
• Compute: 175B parameters means 100× more FLOPs than GPT-2
• Communication: Model parallelism requires high-bandwidth interconnects
• Latency: Sequential generation cannot be parallelized across tokens
• Utilization: Autoregressive generation achieves 20-40\% of peak FLOPS

Instruction Tuning and RLHF

Instruction Tuning

Instruction: Summarize the following in one sentence:
[long text]
Output: [one-sentence summary]

InstructGPT / ChatGPT approach:

1. Pre-train with language modeling
2. Supervised fine-tuning on high-quality instructions
3. Train reward model from human preferences
4. Optimize policy with reinforcement learning

RLHF (Reinforcement Learning from Human Feedback)

Result: Models better aligned with human preferences, more helpful, honest, and harmless.

GPT Capabilities and Limitations

Capabilities

Strong:

• Text generation (creative writing, code, dialogue)
• Translation and summarization
• Question answering
• Few-shot learning
• Chain-of-thought reasoning
• Instruction following

Limitations

Weak:

• Factual accuracy (hallucinations)
• Mathematical reasoning (without tools)
• Long-term coherence in very long texts
• True understanding vs pattern matching
• Consistent personality/beliefs

Hallucinations: Model generates plausible but false information with high confidence.

Mitigation strategies:

• Retrieval-augmented generation (RAG)
• Tool use (calculators, search)
• Verification and fact-checking
• Constitutional AI principles

Exercises

Solutions

Full solutions for all exercises are available at \url{https://deeplearning.hofkensvermeulen.be}.

import torch
import torch.nn as nn
from transformers import GPT2LMHeadModel, GPT2Tokenizer

def compute_lm_loss_teacher_forcing(model, tokenizer, sequence):
    """Compute loss with teacher forcing (standard training)"""
    # Tokenize
    tokens = tokenizer.encode(sequence, return_tensors='pt')
    
    # Input: all tokens except last
    # Target: all tokens except first
    input_ids = tokens[:, :-1]
    target_ids = tokens[:, 1:]
    
    # Forward pass
    outputs = model(input_ids, labels=target_ids)
    loss = outputs.loss
    logits = outputs.logits
    
    # Compute per-token loss
    loss_fct = nn.CrossEntropyLoss(reduction='none')
    per_token_loss = loss_fct(
        logits.view(-1, logits.size(-1)),
        target_ids.view(-1)
    )
    
    return loss, per_token_loss, tokens

def compute_lm_loss_scheduled_sampling(model, tokenizer, sequence, 
                                      sampling_prob=0.5):
    """Compute loss with scheduled sampling (exposure schedule)"""
    tokens = tokenizer.encode(sequence, return_tensors='pt')
    
    total_loss = 0
    per_token_losses = []
    generated_tokens = [tokens[0, 0].item()]  # Start with first token
    
    for i in range(1, tokens.size(1)):
        # Decide: use ground truth or model prediction
        if torch.rand(1).item() < sampling_prob:
            # Use model's own prediction
            input_ids = torch.tensor([generated_tokens]).to(tokens.device)
            with torch.no_grad():
                outputs = model(input_ids)
                next_token = outputs.logits[0, -1, :].argmax().item()
        else:
            # Use ground truth (teacher forcing)
            next_token = tokens[0, i-1].item()
        
        generated_tokens.append(next_token)
        
        # Compute loss for this position
        input_ids = torch.tensor([generated_tokens[:-1]]).to(tokens.device)
        target = tokens[0, i].unsqueeze(0)
        
        outputs = model(input_ids, labels=target)
        per_token_losses.append(outputs.loss.item())
        total_loss += outputs.loss.item()
    
    avg_loss = total_loss / (tokens.size(1) - 1)
    return avg_loss, per_token_losses, generated_tokens

# Example
sequence = "The quick brown fox"
model = GPT2LMHeadModel.from_pretrained('gpt2')
tokenizer = GPT2Tokenizer.from_pretrained('gpt2')
model.eval()

# Teacher forcing
loss_tf, per_token_tf, tokens = compute_lm_loss_teacher_forcing(
    model, tokenizer, sequence
)

# Scheduled sampling
loss_ss, per_token_ss, gen_tokens = compute_lm_loss_scheduled_sampling(
    model, tokenizer, sequence, sampling_prob=0.5
)

print(f"Sequence: {sequence}")
print(f"Tokens: {tokenizer.convert_ids_to_tokens(tokens[0])}")
print(f"\nTeacher forcing loss: {loss_tf.item():.4f}")
print(f"Scheduled sampling loss: {loss_ss:.4f}")

Position 1 (quick): loss = 3.45, prob = 0.032
Position 2 (brown): loss = 4.12, prob = 0.016
Position 3 (fox): loss = 2.87, prob = 0.057
Average loss: 3.48

Position 1: Use GT "The" -> predict "quick" (loss = 3.45)
Position 2: Use prediction "fast" -> predict "brown" (loss = 5.23)
Position 3: Use GT "brown" -> predict "fox" (loss = 2.91)
Average loss: 3.86

from transformers import GPT2LMHeadModel, GPT2Tokenizer
import torch
import numpy as np

def create_few_shot_prompt(examples, test_input, n_shots):
    """Create prompt with n examples"""
    prompt = ""
    
    # Add n examples
    for i in range(n_shots):
        prompt += f"Input: {examples[i]['text']}\n"
        prompt += f"Label: {examples[i]['label']}\n\n"
    
    # Add test input
    prompt += f"Input: {test_input}\n"
    prompt += f"Label:"
    
    return prompt

def predict_with_few_shot(model, tokenizer, prompt, labels=['positive', 'negative']):
    """Predict label using few-shot prompting"""
    # Encode prompt
    input_ids = tokenizer.encode(prompt, return_tensors='pt')
    
    # Generate continuation
    with torch.no_grad():
        outputs = model(input_ids)
        logits = outputs.logits[0, -1, :]  # Last token logits
    
    # Get probabilities for each label
    label_probs = {}
    for label in labels:
        label_tokens = tokenizer.encode(f" {label}", add_special_tokens=False)
        # Use first token of label
        label_id = label_tokens[0]
        label_probs[label] = torch.softmax(logits, dim=-1)[label_id].item()
    
    # Normalize probabilities
    total = sum(label_probs.values())
    label_probs = {k: v/total for k, v in label_probs.items()}
    
    # Return most likely label
    predicted_label = max(label_probs, key=label_probs.get)
    return predicted_label, label_probs

# Example dataset: sentiment classification
train_examples = [
    {"text": "This movie was amazing!", "label": "positive"},
    {"text": "I loved every minute of it.", "label": "positive"},
    {"text": "Terrible waste of time.", "label": "negative"},
    {"text": "Boring and predictable.", "label": "negative"},
    {"text": "Absolutely fantastic!", "label": "positive"},
    {"text": "Worst film I've ever seen.", "label": "negative"},
    {"text": "Brilliant performances.", "label": "positive"},
    {"text": "Completely disappointing.", "label": "negative"},
    {"text": "A masterpiece!", "label": "positive"},
    {"text": "Awful in every way.", "label": "negative"},
]

test_examples = [
    {"text": "Great acting and story.", "label": "positive"},
    {"text": "Not worth watching.", "label": "negative"},
    {"text": "Exceeded my expectations.", "label": "positive"},
    {"text": "Very dull and slow.", "label": "negative"},
    # ... 20 more test examples
]

model = GPT2LMHeadModel.from_pretrained('gpt2')
tokenizer = GPT2Tokenizer.from_pretrained('gpt2')
model.eval()

def evaluate_few_shot(model, tokenizer, train_examples, test_examples, 
                     n_shots_list=[0, 1, 5, 10]):
    """Evaluate accuracy for different numbers of shots"""
    results = {}
    
    for n_shots in n_shots_list:
        correct = 0
        predictions = []
        
        for test_ex in test_examples:
            # Create prompt with n examples
            prompt = create_few_shot_prompt(
                train_examples[:n_shots],
                test_ex['text'],
                n_shots
            )
            
            # Predict
            pred_label, probs = predict_with_few_shot(
                model, tokenizer, prompt
            )
            predictions.append(pred_label)
            
            # Check if correct
            if pred_label == test_ex['label']:
                correct += 1
        
        accuracy = correct / len(test_examples)
        results[n_shots] = {
            'accuracy': accuracy,
            'predictions': predictions
        }
        
        print(f"{n_shots}-shot accuracy: {accuracy:.2
    
    return results

# Run evaluation
results = evaluate_few_shot(
    model, tokenizer, 
    train_examples, test_examples,
    n_shots_list=[0, 1, 5, 10]
)

# Plot results
import matplotlib.pyplot as plt

shots = list(results.keys())
accuracies = [results[s]['accuracy'] for s in shots]

plt.figure(figsize=(10, 6))
plt.plot(shots, accuracies, 'o-', linewidth=2, markersize=10)
plt.xlabel('Number of Examples (Shots)')
plt.ylabel('Accuracy')
plt.title('Few-Shot Learning Performance')
plt.grid(True)
plt.xticks(shots)
plt.ylim([0, 1])

# Add value labels
for x, y in zip(shots, accuracies):
    plt.text(x, y + 0.02, f'{y:.1

plt.savefig('few_shot_performance.png', dpi=150)

import torch
import torch.nn as nn
from torch.utils.data import DataLoader
import numpy as np
import matplotlib.pyplot as plt

def create_model(n_params_target, vocab_size=10000, seq_length=128):
    """Create GPT-style model with approximately n_params_target parameters"""
    # Solve for d_model given target parameters
    # Approximate: P ≈ V*d + L*(12*d^2 + 8*d*d_ff)
    # Assume L=6, d_ff=4*d, V=10000
    
    # Simplified: P ≈ V*d + L*60*d^2
    # Solve quadratic for d
    L = 6
    a = L * 60
    b = vocab_size
    c = -n_params_target
    
    d_model = int((-b + np.sqrt(b**2 - 4*a*c)) / (2*a))
    d_model = max(64, d_model)  # Minimum size
    
    # Create model
    model = nn.TransformerDecoder(
        nn.TransformerDecoderLayer(
            d_model=d_model,
            nhead=max(1, d_model // 64),
            dim_feedforward=4*d_model,
            batch_first=True
        ),
        num_layers=L
    )
    
    # Add embedding and output layers
    embedding = nn.Embedding(vocab_size, d_model)
    output_layer = nn.Linear(d_model, vocab_size)
    
    # Count actual parameters
    total_params = sum(p.numel() for p in model.parameters())
    total_params += sum(p.numel() for p in embedding.parameters())
    total_params += sum(p.numel() for p in output_layer.parameters())
    
    return model, embedding, output_layer, total_params

def train_model(model, embedding, output_layer, train_loader, 
                epochs=50, lr=1e-3):
    """Train model and return final loss"""
    optimizer = torch.optim.Adam(
        list(model.parameters()) + 
        list(embedding.parameters()) + 
        list(output_layer.parameters()),
        lr=lr
    )
    criterion = nn.CrossEntropyLoss()
    
    losses = []
    
    for epoch in range(epochs):
        epoch_loss = 0
        for batch in train_loader:
            input_ids, target_ids = batch
            
            # Forward pass
            x = embedding(input_ids)
            x = model(x, x)  # Self-attention
            logits = output_layer(x)
            
            # Compute loss
            loss = criterion(
                logits.view(-1, logits.size(-1)),
                target_ids.view(-1)
            )
            
            # Backward pass
            optimizer.zero_grad()
            loss.backward()
            optimizer.step()
            
            epoch_loss += loss.item()
        
        avg_loss = epoch_loss / len(train_loader)
        losses.append(avg_loss)
    
    return losses[-1]  # Return final loss

# Train models of different sizes
param_sizes = [10e6, 50e6, 100e6, 500e6]  # 10M, 50M, 100M, 500M
final_losses = []
actual_params = []

for target_params in param_sizes:
    print(f"\nTraining model with ~{target_params/1e6:.0f}M parameters...")
    
    # Create model
    model, emb, out, n_params = create_model(target_params)
    actual_params.append(n_params)
    print(f"Actual parameters: {n_params/1e6:.1f}M")
    
    # Train model
    final_loss = train_model(model, emb, out, train_loader, epochs=50)
    final_losses.append(final_loss)
    print(f"Final loss: {final_loss:.4f}")

# Plot on log-log scale
plt.figure(figsize=(10, 6))
plt.loglog(actual_params, final_losses, 'o-', linewidth=2, markersize=10)
plt.xlabel('Parameters (log scale)')
plt.ylabel('Loss (log scale)')
plt.title('Scaling Law: Loss vs Model Size')
plt.grid(True, which='both', alpha=0.3)

# Fit power law: L = a * N^(-b)
log_params = np.log(actual_params)
log_losses = np.log(final_losses)
coeffs = np.polyfit(log_params, log_losses, 1)
exponent = -coeffs[0]
intercept = coeffs[1]

# Plot fitted line
params_fit = np.logspace(np.log10(min(actual_params)), 
                         np.log10(max(actual_params)), 100)
losses_fit = np.exp(intercept) * params_fit**(-exponent)
plt.loglog(params_fit, losses_fit, '--', label=f'Power law fit: L ∝ N^{-exponent:.3f}')

plt.legend()
plt.savefig('scaling_law.png', dpi=150)

print(f"\nScaling law exponent: {exponent:.3f}")
print(f"Power law: L = {np.exp(intercept):.2f} * N^(-{exponent:.3f})")