T5 and BART: Encoder-Decoder Architectures

Chapter Overview

T5 (Text-to-Text Transfer Transformer) and BART (Bidirectional and Auto-Regressive Transformers) represent encoder-decoder architectures that combine the strengths of BERT and GPT. This chapter covers their architectures, pre-training objectives, unified text-to-text framework, and applications to sequence-to-sequence tasks.

Learning Objectives

1. Understand encoder-decoder transformer architectures
2. Implement span corruption and denoising objectives
3. Apply text-to-text framework to diverse tasks
4. Compare T5, BART, and other seq2seq transformers
5. Fine-tune for summarization, translation, and question answering
6. Understand prefix LM and mixture of denoisers

T5: Text-to-Text Transfer Transformer

Unified Text-to-Text Framework

T5 introduces a conceptually elegant framework that reformulates every NLP task as text-to-text transformation. Rather than designing task-specific architectures with classification heads, span prediction layers, or other specialized output structures, T5 treats all tasks uniformly: the model receives text as input and produces text as output. This unification enables a single model architecture and training objective to handle diverse tasks ranging from translation and summarization to classification and question answering.

The text-to-text framework operates by prepending task-specific prefixes to the input text. For translation, the input becomes "translate English to German: That is good", and the model generates "Das ist gut". For summarization, the input is "summarize: [article text]", and the model produces a concise summary. Even classification tasks, which traditionally output discrete labels, are reformulated as text generation: "sst2 sentence: This movie is great" produces the text "positive" rather than a class index. Question answering similarly becomes "question: What is the capital of France? context: Paris is the capital and largest city of France..." with the model generating "Paris" as output.

This unification provides several compelling advantages. First, a single model can handle all tasks without architectural modifications, simplifying deployment and maintenance. Second, the same pre-training objective and fine-tuning procedure apply across tasks, eliminating the need for task-specific training strategies. Third, the framework enables natural transfer learning across tasks—knowledge learned from translation can potentially benefit summarization, and vice versa. Fourth, evaluation becomes consistent across tasks, as all outputs are text sequences that can be compared using standard metrics. The text-to-text framework represents a philosophical shift toward treating language understanding and generation as a unified capability rather than separate skills requiring different architectures.

T5 Architecture

T5 employs a standard encoder-decoder transformer architecture with several important modifications that distinguish it from the original transformer design. The architecture combines the bidirectional encoding capabilities of BERT with the autoregressive generation capabilities of GPT, creating a model that excels at both understanding input context and generating coherent output sequences.

The encoder processes the input text using fully-visible self-attention, identical to BERT's architecture. Each token in the encoder can attend to all other tokens in the input sequence without any causal masking, enabling the model to build rich bidirectional representations that capture both left and right context. This bidirectional attention is crucial for understanding tasks where the meaning of each token depends on the entire input context. The encoder consists of a stack of transformer layers, each containing multi-head self-attention followed by a position-wise feed-forward network, with residual connections and layer normalization applied in the pre-norm configuration for improved training stability.

The decoder generates the output text autoregressively using causal self-attention, similar to GPT's architecture. Each position in the decoder can only attend to previous positions in the output sequence, ensuring that the model cannot "cheat" by looking at future tokens during generation. Critically, the decoder also includes cross-attention layers that attend to the encoder's output representations. This cross-attention mechanism allows the decoder to focus on relevant parts of the input sequence while generating each output token, enabling the model to perform sequence-to-sequence transformations like translation and summarization where the output depends heavily on specific input content.

T5's most distinctive architectural innovation is its use of relative positional encodings rather than the absolute sinusoidal or learned positional embeddings used in BERT and GPT. Instead of adding position-specific embeddings to the input, T5 computes position-dependent biases that are added to the attention scores. These biases depend only on the relative distance between query and key positions, not their absolute positions in the sequence. The relative position biases are learned during training and shared across all layers, reducing the number of parameters while providing the model with flexible position information. The biases use a bucketing scheme where nearby positions have unique biases but distant positions share biases, reflecting the intuition that precise relative position matters more for nearby tokens than distant ones.

Pre-Training Objective: Span Corruption

T5 introduces span corruption as its primary pre-training objective, a more sophisticated variant of masked language modeling that better aligns with sequence-to-sequence tasks. Rather than masking individual tokens independently as in BERT, span corruption masks contiguous sequences of tokens and trains the model to predict the entire masked span. This objective encourages the model to learn longer-range dependencies and develop stronger generation capabilities, as the decoder must produce multi-token sequences rather than single tokens.

The span corruption procedure begins by sampling span lengths from a Poisson distribution with parameter $\lambda = 3$, yielding an average span length of 3 tokens. The algorithm then selects spans to mask such that approximately 15\% of tokens in the sequence are corrupted, matching BERT's masking rate for fair comparison. Each masked span is replaced with a unique sentinel token (denoted , , , etc.), which serves as a placeholder indicating that tokens have been removed at this position. The model must predict the original content of each masked span in the correct order, identified by the sentinel tokens.

The training format differs significantly from BERT's masked language modeling. The encoder receives the corrupted input sequence with sentinel tokens replacing the masked spans. The decoder must generate a sequence containing the sentinel tokens followed by the original content of each span. For example, if the original text is "Thank you for inviting me to your party last week" and spans at positions 3-4 and 8-9 are masked, the encoder input becomes "Thank you inviting me to your week". The decoder target is " for party last ", where marks the end of the sequence. This format trains the decoder to produce structured output with clear delimiters, a skill that transfers well to downstream generation tasks.

The computational efficiency of span corruption is notable. By masking spans rather than individual tokens, the number of prediction targets decreases while maintaining the same fraction of corrupted tokens. If 15\% of tokens are masked in spans of average length 3, only 5\% of positions contain sentinel tokens that trigger predictions. This reduces the decoder's generation length compared to predicting every masked token individually, accelerating training. However, the decoder must still generate all the masked tokens, so the total number of tokens predicted remains approximately 15\% of the input length. The efficiency gain comes from the reduced number of sentinel tokens that must be processed by the encoder.

The span corruption objective provides several advantages over BERT's masked language modeling for encoder-decoder models. First, it trains the decoder to generate multi-token sequences, developing the autoregressive generation capabilities needed for downstream tasks like summarization and translation. Second, it encourages the model to learn longer-range dependencies, as predicting a span requires understanding the broader context rather than just neighboring tokens. Third, it creates a more challenging task that prevents the model from relying on simple local patterns, forcing it to develop deeper semantic understanding. Fourth, the sentinel token mechanism provides a natural way to structure the decoder's output, which transfers to tasks requiring structured generation.

Thank you  me to your  week

 for inviting  party last 

T5 Model Sizes and Scaling

T5 was released in five sizes to accommodate different computational budgets and performance requirements:

T5 Model Sizes:

• T5-Small: 60M parameters, 6 enc + 6 dec layers, $d=512$, 8 heads
  • Memory: 240 MB (FP32), 120 MB (FP16)
  • Training: 2-3 days on 8 GPUs, cost $\approx$ \$2,000
• T5-Base: 220M parameters, 12 enc + 12 dec layers, $d=768$, 12 heads
  • Memory: 880 MB (FP32), 440 MB (FP16)
  • Training: 1 week on 64 TPU cores, cost $\approx$ \$10,000-\$15,000
• T5-Large: 770M parameters, 24 enc + 24 dec layers, $d=1024$, 16 heads
  • Memory: 3 GB (FP32), 1.5 GB (FP16)
  • Training: 2-3 weeks on 256 TPU cores, cost $\approx$ \$50,000-\$75,000
• T5-3B: 3 billion parameters, 24 enc + 24 dec layers, $d=1024$, 32 heads
  • Memory: 12 GB (FP32), 6 GB (FP16)
  • Training: 1 month on 512 TPU cores, cost $>$ \$200,000
• T5-11B: 11 billion parameters, 24 enc + 24 dec layers, $d=1024$, 128 heads
  • Memory: 44 GB (FP32), 22 GB (FP16)
  • Training: 2-3 months on 1024 TPU cores, cost $>$ \$1,000,000

The scaling behavior reveals important insights about encoder-decoder architectures. Performance improves consistently with model size, but the rate of improvement decreases at larger scales. The cost per percentage point of accuracy improvement increases dramatically beyond T5-3B, suggesting that for most practical applications, T5-Base or T5-Large provide the best trade-off between performance and computational cost. The largest models are primarily valuable for research into scaling laws and for applications where even small accuracy improvements justify substantial computational investment.

T5 Training Details

T5's pre-training represents a massive computational undertaking that required careful optimization of hardware utilization and training procedures. The model was trained on the Colossal Clean Crawled Corpus (C4), a dataset of approximately 750 GB of cleaned English text extracted from Common Crawl. The C4 dataset underwent extensive filtering to remove low-quality content, including deduplication, language identification to retain only English text, removal of placeholder text and profanity, and filtering of sentences without terminal punctuation. This cleaning process reduced the raw Common Crawl data by approximately 90\%, but the resulting corpus provided much higher quality training signal.

The training infrastructure for T5-11B, the largest variant, required 1024 TPU v3 cores running continuously for approximately 2-3 months. Each TPU v3 core provides roughly 123 TFLOPS of bfloat16 performance, yielding a combined peak performance of approximately 126 PFLOPS for the full training cluster. The training used a batch size of 2048 sequences, each of maximum length 512 tokens, for a total of 1,048,576 tokens per batch. This enormous batch size enabled efficient utilization of the TPU hardware and provided stable gradient estimates despite the model's scale. The learning rate schedule employed a linear warmup over 10,000 steps to a peak learning rate of $10^{-2}$, followed by inverse square root decay. The high peak learning rate, much larger than typical for transformer training, was enabled by the large batch size and careful gradient clipping.

The computational cost of T5-11B training is staggering. With 11 billion parameters and processing approximately 1 trillion tokens during training (the C4 dataset seen roughly 1.3 times), the total computation exceeds $10^{24}$ FLOPs. At an effective compute rate of 50 PFLOPS (assuming 40\% utilization of the 126 PFLOPS peak), the training requires approximately $10^{24} / (50 \times 10^{15}) = 20$ million seconds, or roughly 230 days of continuous computation. The reported 2-3 month training time suggests either higher utilization rates or more efficient training procedures than this conservative estimate. The estimated cost exceeds \$1 million using cloud TPU pricing, making T5-11B one of the most expensive models trained at the time of its release in 2019.

T5-Base training is far more accessible, requiring approximately 1 week on 64 TPU v3 cores (128 TPU cores total). The batch size is reduced to 128 sequences of 512 tokens, totaling 65,536 tokens per batch. The training processes approximately 34 billion tokens (the C4 dataset seen once), requiring roughly $10^{21}$ FLOPs total. At an effective compute rate of 2 PFLOPS, the training takes approximately 5-7 days, matching the reported training time. The estimated cost is \$10,000-\$15,000, making T5-Base training feasible for well-funded academic labs and smaller companies. The more modest computational requirements have enabled widespread experimentation with the T5 architecture and training approach.

The memory requirements during training are substantial due to the encoder-decoder architecture. For T5-11B with batch size 2048 and sequence length 512, the activations alone consume approximately 200-300 GB of memory. The model parameters require 44 GB in FP32, and the optimizer states (Adam maintains first and second moment estimates) require an additional 88 GB. The total memory footprint exceeds 400 GB, necessitating model parallelism across multiple TPU cores. The training employed a combination of data parallelism (different sequences on different cores) and model parallelism (different layers on different cores) to distribute the memory and computation efficiently. The cross-attention mechanism in the decoder requires storing encoder outputs for all sequences in the batch, adding significant memory overhead compared to encoder-only or decoder-only architectures.

T5-11B Training Configuration:

• Hardware: 1024 TPU v3 cores ($\approx$ 126 PFLOPS peak)
• Training time: 2-3 months continuous
• Dataset: C4 (750 GB cleaned text, $\approx$ 1 trillion tokens)
• Batch size: 2048 sequences $\times$ 512 tokens = 1,048,576 tokens/batch
• Learning rate: $10^{-2}$ peak with inverse square root decay
• Total computation: $> 10^{24}$ FLOPs
• Estimated cost: $>$ \$1,000,000
• Memory: $>$ 400 GB (requires model parallelism)

T5-Base Training Configuration:

• Hardware: 64 TPU v3 chips (128 cores, $\approx$ 15 PFLOPS peak)
• Training time: 5-7 days
• Dataset: C4 (750 GB, single pass $\approx$ 34 billion tokens)
• Batch size: 128 sequences $\times$ 512 tokens = 65,536 tokens/batch
• Learning rate: $10^{-2}$ peak with inverse square root decay
• Total computation: $\approx 10^{21}$ FLOPs
• Estimated cost: \$10,000-\$15,000
• Memory: $\approx$ 20-30 GB (fits on single GPU with gradient accumulation)

The training procedures incorporated several optimizations to improve efficiency and stability. Mixed precision training with bfloat16 reduced memory consumption and accelerated computation on TPU hardware. Gradient clipping prevented instability from occasional large gradients. Dropout was applied with rate 0.1 during pre-training to prevent overfitting, though later work (T5.1.1) found that removing dropout during pre-training improved performance. The relative position biases were initialized to small random values and learned during training, converging to patterns that emphasized nearby positions while maintaining some attention to distant positions.

BART: Denoising Autoencoder

BART Architecture and Design Philosophy

BART (Bidirectional and Auto-Regressive Transformers) represents Facebook AI Research's approach to combining the strengths of BERT and GPT through a denoising autoencoder framework. While T5 focuses on the text-to-text paradigm with task-specific prefixes, BART emphasizes learning robust representations through diverse corruption strategies during pre-training. The model architecture is conceptually similar to T5—an encoder-decoder transformer—but the pre-training approach and design philosophy differ significantly.

The BART encoder employs fully bidirectional attention identical to BERT, allowing each token to attend to all other tokens in the input sequence. This bidirectional processing enables the encoder to build rich contextual representations that capture dependencies in both directions. The encoder processes corrupted input text, where corruption can take many forms including token masking, deletion, infilling, sentence permutation, or document rotation. The diversity of corruption strategies forces the encoder to learn robust representations that can handle various types of noise and structural perturbations.

The BART decoder uses causal self-attention like GPT, generating output tokens autoregressively from left to right. Each position in the decoder can only attend to previous positions in the output sequence, maintaining the autoregressive property essential for text generation. The decoder also includes cross-attention layers that attend to the encoder's output representations, enabling it to focus on relevant parts of the corrupted input while reconstructing the original text. This cross-attention mechanism is crucial for tasks like summarization and translation where the output must be grounded in specific input content.

BART-large, the primary configuration, uses 12 encoder layers and 12 decoder layers with hidden dimension $d = 1024$ and 16 attention heads. This configuration is comparable to BERT-large in terms of depth and width, but the encoder-decoder architecture results in more total parameters. The model uses learned absolute positional embeddings rather than T5's relative position biases or the original transformer's sinusoidal encodings. The vocabulary contains approximately 50,000 tokens using byte-pair encoding (BPE), providing finer-grained tokenization than T5's 32,000-token SentencePiece vocabulary.

BART Parameter Breakdown and Memory Requirements

Understanding BART-large's parameter distribution reveals how the model allocates capacity across its components. Each encoder layer contains approximately 12.6 million parameters. The self-attention mechanism requires four projection matrices ($\mW^Q$, $\mW^K$, $\mW^V$, $\mW^O$), each of dimension $1024 \times 1024$, contributing $4 \times 1024^2 = 4{,}194{,}304$ parameters. The feed-forward network uses expansion factor 4, projecting from 1024 to 4096 dimensions and back, contributing $2 \times 1024 \times 4096 = 8{,}388{,}608$ parameters. Layer normalization adds minimal parameters. Multiplying by 12 encoder layers yields approximately 151 million parameters in the encoder stack.

Each decoder layer contains approximately 16.8 million parameters due to the additional cross-attention mechanism. The causal self-attention contributes 4.2 million parameters, identical to the encoder's self-attention. The cross-attention layer adds another 4.2 million parameters for its query, key, value, and output projections. The feed-forward network contributes 8.4 million parameters, same as the encoder. Multiplying by 12 decoder layers yields approximately 202 million parameters in the decoder stack. The token embeddings add $50{,}000 \times 1024 = 51{,}200{,}000$ parameters, and positional embeddings for sequences up to 1024 tokens add another $1024 \times 1024 = 1{,}048{,}576$ parameters. The total reaches approximately 406 million parameters. This analysis follows the same component-level methodology used for BERT-base in Section~[ref].

The memory requirements for BART-large are substantial. In FP32, the 406 million parameters occupy $406{,}000{,}000 \times 4 = 1{,}624$ MB, or approximately 1.6 GB. Mixed precision training with FP16 activations and FP32 master weights reduces the working memory to approximately 812 MB for the model parameters during forward and backward passes. For inference, pure FP16 weights require only 812 MB, enabling BART-large to run on GPUs with 12-16 GB of memory with reasonable batch sizes. Training with batch size 32 and sequence length 512 requires approximately 20-25 GB of GPU memory, necessitating high-memory GPUs like the V100 (32 GB) or A100 (40-80 GB).

BART-large Configuration:

• Encoder: 12 layers, Decoder: 12 layers
• Hidden: $d = 1024$, Heads: $h = 16$, FFN: $d_{ff} = 4096$
• Vocabulary: $V \approx 50{,}000$ (BPE)
• Parameters: $\approx 406$M

Parameter breakdown:

Memory requirements:

• FP32: 1.6 GB (model parameters only)
• FP16: 812 MB (inference)
• Training (mixed precision, batch size 32, sequence length 512): $\approx$ 20-25 GB

Denoising Objectives and Corruption Strategies

BART's key innovation lies in exploring multiple corruption strategies during pre-training, systematically evaluating which types of noise lead to the most robust and transferable representations. Unlike BERT's single masking strategy or T5's span corruption, BART experiments with five different corruption approaches and combinations thereof. This exploration revealed that the choice of corruption strategy significantly impacts downstream task performance, with different strategies providing complementary benefits.

Token masking, borrowed directly from BERT, replaces random tokens with a special [MASK] token. Approximately 15\% of tokens are selected and replaced, forcing the model to predict the original tokens based on surrounding context. This strategy is familiar and well-understood, providing a baseline for comparison with other corruption approaches. However, token masking has limitations: the [MASK] token never appears during fine-tuning, creating a train-test mismatch, and the independent masking of tokens doesn't encourage the model to learn longer-range dependencies or sequential generation capabilities.

Token deletion removes random tokens entirely from the input sequence, forcing the model to determine which positions are missing and what content should fill them. Unlike masking, which provides explicit markers indicating where tokens were removed, deletion requires the model to infer the locations of missing content from the remaining context. This creates a more challenging task that encourages the model to develop robust positional understanding and the ability to detect gaps in the input. For example, deleting "B" and "D" from "A B C D E" yields "A C E", and the model must reconstruct the full sequence "A B C D E" without explicit indicators of where tokens were removed.

Text infilling represents a more sophisticated corruption strategy that combines aspects of span masking and deletion. Spans of text are sampled (with lengths drawn from a Poisson distribution with $\lambda = 3$, similar to T5), but instead of replacing each span with a unique sentinel token, all spans are replaced with a single [MASK] token. This forces the decoder to determine how many tokens to generate for each masked span based on context alone. For example, replacing "B C D E" in "A B C D E F" with a single [MASK] yields "A [MASK] F", and the model must generate "B C D E" without knowing in advance that four tokens are needed. This uncertainty makes text infilling substantially more challenging than T5's span corruption with explicit sentinel tokens.

Sentence permutation shuffles the order of sentences within a document, requiring the model to reconstruct the original sentence order. This corruption strategy targets document-level structure rather than token-level content, encouraging the model to learn discourse coherence and inter-sentence dependencies. For example, a document with sentences [S1, S2, S3, S4] might be permuted to [S3, S1, S4, S2], and the model must generate the original order [S1, S2, S3, S4]. This task is particularly relevant for summarization and document understanding, where maintaining coherent structure is crucial.

Document rotation selects a random token as the new start of the document and rotates the entire sequence accordingly. The model must identify the true start of the document and generate the original sequence. For example, rotating "A B C D E" at position 3 yields "D E A B C", and the model must recognize that "A" is the true start and generate "A B C D E". This task encourages the model to learn document-level structure and identify natural boundaries, though it proved less effective than other corruption strategies in practice.

The BART paper systematically evaluated these corruption strategies individually and in combination, finding that text infilling combined with sentence permutation provided the best performance across downstream tasks. This combination balances token-level and document-level corruption, encouraging the model to learn both local language patterns and global document structure. The text infilling component develops strong generation capabilities by forcing the model to produce variable-length spans, while sentence permutation develops discourse understanding by requiring the model to reason about inter-sentence relationships.

BART Corruption Strategies:

1. Token Masking: Replace tokens with [MASK] (like BERT)

• 15\% of tokens replaced with [MASK]
• Provides explicit markers for missing content
• Baseline strategy for comparison

Original: A B C D E
Corrupted: A C E
Target: A B C D E

• Model must infer locations of missing tokens
• More challenging than masking
• Encourages robust positional understanding

Original: A B C D E F
Corrupted: A [MASK] F
Target: B C D E

• Span lengths sampled from Poisson($\lambda=3$)
• Model must determine span length from context
• More challenging than T5's sentinel-based span corruption

4. Sentence Permutation: Shuffle sentence order

• Targets document-level structure
• Encourages learning of discourse coherence
• Particularly beneficial for summarization

5. Document Rotation: Rotate document, model finds start

• Less effective than other strategies
• Encourages learning of document boundaries

Best combination (BART's final): Text infilling + sentence permutation

• Balances token-level and document-level corruption
• Develops both generation and discourse understanding
• Achieves best performance across diverse downstream tasks

The cat sat on the mat. It was very comfortable.
The dog barked loudly.

The dog barked loudly.
The [MASK] comfortable.

BART Training Details

BART-large was trained on a combination of datasets totaling approximately 160 GB of text, including BooksCorpus, English Wikipedia, CC-News, OpenWebText, and Stories. This diverse training corpus provides broad coverage of topics and writing styles, enabling the model to learn robust representations that transfer well to downstream tasks. The training used 256 NVIDIA V100 GPUs for approximately 2 weeks, with an estimated cost of \$50,000-\$75,000 using cloud computing resources.

The training configuration employed a batch size of 128 sequences with maximum length 1024 tokens, totaling 131,072 tokens per batch. This large batch size enabled stable training with the Adam optimizer and efficient GPU utilization. The learning rate schedule used a polynomial decay from a peak learning rate of $3 \times 10^{-4}$ with 500 warmup steps. The training processed approximately 50 billion tokens total, seeing the training corpus roughly once. Mixed precision training with FP16 reduced memory consumption and accelerated computation on the V100 GPUs.

The memory requirements during training are substantial due to the encoder-decoder architecture and large batch size. With batch size 128 and sequence length 1024, the activations consume approximately 40-50 GB of memory. The model parameters require 1.6 GB in FP32, and the Adam optimizer states require an additional 3.2 GB. The total memory footprint reaches approximately 50-60 GB, necessitating data parallelism across multiple GPUs. Each GPU processes a subset of the batch, with gradients synchronized across GPUs after each backward pass.

BART-large Training Configuration:

• Hardware: 256 NVIDIA V100 GPUs (32 GB each)
• Training time: $\approx$ 2 weeks
• Dataset: 160 GB text (BooksCorpus, Wikipedia, CC-News, OpenWebText, Stories)
• Batch size: 128 sequences $\times$ 1024 tokens = 131,072 tokens/batch
• Learning rate: $3 \times 10^{-4}$ peak with polynomial decay
• Total tokens: $\approx$ 50 billion
• Estimated cost: \$50,000-\$75,000
• Memory per GPU: $\approx$ 25-30 GB (data parallelism across GPUs)

Encoder-Decoder Efficiency Analysis

Computational Cost of Cross-Attention

Understanding the computational and memory costs of encoder-decoder architectures compared to encoder-only (BERT) or decoder-only (GPT) models is essential for choosing the appropriate architecture for a given task. The key difference lies in the cross-attention mechanism, which enables the decoder to attend to encoder outputs but introduces additional computational and memory overhead.

The cross-attention mechanism in each decoder layer requires computing attention between decoder queries and encoder keys/values. For a decoder sequence of length $n_{\text{dec}}$ and encoder sequence of length $n_{\text{enc}}$, the cross-attention computation involves three main steps. First, the decoder hidden states are projected to queries $\mQ \in \R^{n_{\text{dec}} \times d}$ using weight matrix $\mW^Q \in \R^{d \times d}$, requiring $n_{\text{dec}} \times d^2$ FLOPs. Second, the encoder outputs are projected to keys $\mK \in \R^{n_{\text{enc}} \times d}$ and values $\mV \in \R^{n_{\text{enc}} \times d}$ using weight matrices $\mW^K, \mW^V \in \R^{d \times d}$, requiring $2 \times n_{\text{enc}} \times d^2$ FLOPs. Third, the attention scores $\mS = \mQ \mK\transpose$ are computed, requiring $n_{\text{dec}} \times n_{\text{enc}} \times d$ FLOPs, followed by softmax and multiplication with values, requiring another $n_{\text{dec}} \times n_{\text{enc}} \times d$ FLOPs.

The total computational cost of cross-attention per layer is approximately $n_{\text{dec}} \times d^2 + 2 \times n_{\text{enc}} \times d^2 + 2 \times n_{\text{dec}} \times n_{\text{enc}} \times d$ FLOPs. For typical sequence lengths where $n_{\text{dec}} \approx n_{\text{enc}} = n$, this simplifies to $3nd^2 + 2n^2d$ FLOPs. Comparing to self-attention, which requires $4nd^2 + 2n^2d$ FLOPs, cross-attention adds approximately 75\% of the cost of self-attention per layer. With $L_{\text{dec}}$ decoder layers, the total cross-attention cost is $L_{\text{dec}} \times (3nd^2 + 2n^2d)$ FLOPs.

For T5-Base with 12 decoder layers, $d = 768$, and $n = 512$, the cross-attention computation requires approximately $12 \times (3 \times 512 \times 768^2 + 2 \times 512^2 \times 768) \approx 12 \times (9.1 + 4.0) \times 10^8 = 1.57 \times 10^{11}$ FLOPs per forward pass. This represents approximately 15-20\% of the total forward pass computation, a significant but not dominant fraction. The cross-attention cost scales linearly with the number of decoder layers and quadratically with sequence length, making it increasingly expensive for long sequences.

The memory requirements for cross-attention are equally important. The encoder outputs must be stored in memory for all decoder layers to access during cross-attention. For batch size $B$, encoder sequence length $n_{\text{enc}}$, and hidden dimension $d$, the encoder outputs require $B \times n_{\text{enc}} \times d$ values. For T5-Base with batch size 32, sequence length 512, and dimension 768, this amounts to $32 \times 512 \times 768 = 12{,}582{,}912$ values, or approximately 50 MB in FP32 or 25 MB in FP16. While modest compared to model parameters, this memory scales linearly with batch size and sequence length, becoming significant for large batches or long sequences.

Additionally, the cross-attention mechanism requires storing attention weights $\mA \in \R^{n_{\text{dec}} \times n_{\text{enc}}}$ for each head in each layer during training (for backpropagation). With $h$ attention heads and $L_{\text{dec}}$ decoder layers, the total attention weight memory is $B \times L_{\text{dec}} \times h \times n_{\text{dec}} \times n_{\text{enc}}$ values. For T5-Base with batch size 32, 12 decoder layers, 12 heads, and sequence length 512, this amounts to $32 \times 12 \times 12 \times 512 \times 512 = 1{,}207{,}959{,}552$ values, or approximately 4.8 GB in FP32 or 2.4 GB in FP16. This memory requirement can become a bottleneck for training with large batch sizes or long sequences.

Cross-attention computational cost per layer:

For $n_{\text{dec}} = n_{\text{enc}} = n$:

Memory requirements:

• Encoder outputs: $B \times n_{\text{enc}} \times d$ values (must be stored for all decoder layers)
• Cross-attention weights (training): $B \times L_{\text{dec}} \times h \times n_{\text{dec}} \times n_{\text{enc}}$ values

Example: T5-Base (batch size 32, sequence length 512):

• Cross-attention FLOPs per layer: $\approx 1.3 \times 10^{10}$ FLOPs
• Total cross-attention (12 layers): $\approx 1.6 \times 10^{11}$ FLOPs (15-20\% of forward pass)
• Encoder output memory: 50 MB (FP32) or 25 MB (FP16)
• Cross-attention weight memory: 4.8 GB (FP32) or 2.4 GB (FP16)

Comparison: Encoder-Decoder vs Decoder-Only

The choice between encoder-decoder architectures (T5, BART) and decoder-only architectures (GPT) involves fundamental trade-offs in computational efficiency, memory usage, and task suitability. Understanding these trade-offs is essential for practitioners deciding which architecture to use for their specific application.

Decoder-only models like GPT use only causal self-attention, processing sequences autoregressively from left to right. For a sequence of length $n$, a decoder-only model with $L$ layers requires approximately $L \times (4nd^2 + 2n^2d)$ FLOPs for the forward pass. The memory requirements include model parameters, activations, and KV cache for generation. For GPT-2 with 12 layers, $d = 768$, and $n = 512$, the forward pass requires approximately $12 \times (4 \times 512 \times 768^2 + 2 \times 512^2 \times 768) \approx 1.2 \times 10^{12}$ FLOPs. The KV cache for generation requires $2 \times L \times n \times d$ values, or approximately 75 MB in FP32 for GPT-2 with sequence length 1024.

Encoder-decoder models like T5 and BART use separate encoder and decoder stacks with cross-attention connecting them. For input sequence length $n_{\text{enc}}$ and output sequence length $n_{\text{dec}}$, the encoder requires $L_{\text{enc}} \times (4n_{\text{enc}}d^2 + 2n_{\text{enc}}^2d)$ FLOPs, and the decoder requires $L_{\text{dec}} \times (4n_{\text{dec}}d^2 + 2n_{\text{dec}}^2d + 3n_{\text{dec}}d^2 + 2n_{\text{dec}}n_{\text{enc}}d)$ FLOPs. For T5-Base with $n_{\text{enc}} = n_{\text{dec}} = 512$, the total forward pass requires approximately $2.1 \times 10^{12}$ FLOPs, roughly 1.75× more than GPT-2 of similar size. The memory requirements include encoder outputs ($B \times n_{\text{enc}} \times d$) and cross-attention weights, adding 25-50 MB beyond decoder-only models.

The parameter count comparison reveals that encoder-decoder models require more parameters than decoder-only models of similar capacity. T5-Base with 220 million parameters has 12 encoder layers (85M parameters) and 12 decoder layers (113M parameters including cross-attention). GPT-2 with 12 layers and the same hidden dimension contains only 117 million parameters, as it lacks the encoder stack and cross-attention mechanisms. This means encoder-decoder models require approximately 1.9× more parameters than decoder-only models with the same number of layers and hidden dimension.

However, the computational comparison depends critically on the task. For generation tasks where the input is short and the output is long (e.g., generating a long document from a short prompt), decoder-only models can be more efficient. The encoder-decoder model processes the short input once through the encoder, then generates the long output through the decoder with cross-attention. The decoder-only model must process the entire sequence (input plus generated output) autoregressively, with each new token requiring attention over all previous tokens. For input length $n_{\text{in}}$ and output length $n_{\text{out}}$, the decoder-only model requires $\sum_{t=1}^{n_{\text{out}}} (n_{\text{in}} + t) \approx n_{\text{out}} \times n_{\text{in}} + n_{\text{out}}^2/2$ attention operations, while the encoder-decoder model requires $n_{\text{in}}^2$ (encoder) plus $n_{\text{out}}^2$ (decoder self-attention) plus $n_{\text{out}} \times n_{\text{in}}$ (cross-attention). When $n_{\text{out}} \gg n_{\text{in}}$, the encoder-decoder model is more efficient.

For tasks where the input is long and the output is short (e.g., classification or extractive question answering), decoder-only models can be more efficient. The encoder-decoder model must process the long input through the encoder, then generate the short output through the decoder. The decoder-only model processes the input once, then generates the short output. However, encoder-only models like BERT are typically most efficient for these tasks, as they avoid the decoder entirely and use a simple classification head.

The memory efficiency comparison favors decoder-only models for inference, as they avoid storing encoder outputs and cross-attention weights. However, for training with large batch sizes, the difference is less significant, as both architectures require substantial memory for activations and gradients. The KV cache for decoder-only models grows with the total sequence length (input plus output), while encoder-decoder models cache only decoder states, potentially providing memory advantages for long input sequences.

When to use encoder-decoder (T5, BART):

• Sequence-to-sequence tasks: translation, summarization, question answering with generation
• Tasks requiring bidirectional understanding of input: the encoder can attend to the full input context
• Tasks with long input and short output: encoder processes input once, decoder generates short output
• Multi-task learning: text-to-text framework enables unified training across diverse tasks

When to use decoder-only (GPT):

• Pure generation tasks: story generation, dialogue, code generation
• Tasks with short input and long output: decoder-only can be more efficient
• In-context learning: decoder-only models excel at few-shot learning from examples in the prompt
• Simplicity: decoder-only architecture is simpler to implement and deploy

Computational comparison (similar capacity):

• Parameters: Encoder-decoder $\approx$ 1.9× decoder-only (due to encoder stack and cross-attention)
• FLOPs per forward pass: Encoder-decoder $\approx$ 1.5-2× decoder-only (depends on sequence lengths)
• Memory (inference): Decoder-only more efficient (no encoder outputs or cross-attention weights)
• Memory (training): Similar for both architectures with large batch sizes

Comparing T5 and BART

Performance comparison on GLUE:

• T5-11B: 90.3 (state-of-art at release)
• BART-large: 88.4
• RoBERTa-large: 88.5

Summarization (CNN/DailyMail):

• BART-large: ROUGE-L 44.16 (best)
• T5-base: ROUGE-L 42.05

Prefix Language Models

Prefix LM Objective

Prefix: "Translate to French: Hello"
Target: "Bonjour"

Attention mask:

• Prefix tokens can attend to all prefix
• Target tokens attend causally
• Enables both understanding and generation

Models using Prefix LM:

• UniLM (Microsoft)
• GLM (Tsinghua)
• UL2 (Google)

Applications and Fine-tuning

Summarization

Task: Input document $\to$ Summary

summarize: [article text]

BART approach:

• Encoder: Full article
• Decoder: Generate summary

Metrics:

• ROUGE-1, ROUGE-2, ROUGE-L (n-gram overlap)
• BERTScore (semantic similarity)

Translation

translate English to German: That is good.

Output: "Das ist gut."

Multi-task advantage: Single T5 model handles multiple language pairs by conditioning on task prefix.

Question Answering

question: What is the capital of France?
context: Paris is the capital and largest city of France...

Output: "Paris"

Comparison to BERT:

• BERT: Span prediction (start/end positions)
• T5: Text generation (more flexible)

Mixture of Denoisers (UL2)

UL2 combines multiple objectives:

R-Denoiser (Regular): Short spans (like T5)

S-Denoiser (Sequential): Prefix LM

X-Denoiser (Extreme): Very long spans or high corruption

Benefits:

• More robust representations
• Better transfer to diverse tasks
• Single model for understanding and generation

Exercises

Solutions

Full solutions for all exercises are available at \url{https://deeplearning.hofkensvermeulen.be}.

import numpy as np
import random

def sample_span_lengths(num_spans, lambda_param=3):
    """Sample span lengths from Poisson distribution"""
    lengths = np.random.poisson(lambda_param, num_spans)
    # Ensure minimum length of 1
    lengths = np.maximum(lengths, 1)
    return lengths

def corrupt_with_spans(text, corruption_rate=0.15, lambda_param=3):
    """Implement T5 span corruption"""
    tokens = text.split()
    n_tokens = len(tokens)
    
    # Calculate number of tokens to corrupt
    n_corrupt = int(n_tokens * corruption_rate)
    
    # Sample number of spans (average span length = lambda_param)
    n_spans = max(1, n_corrupt // lambda_param)
    
    # Sample span lengths
    span_lengths = sample_span_lengths(n_spans, lambda_param)
    
    # Adjust if total exceeds n_corrupt
    while sum(span_lengths) > n_corrupt:
        span_lengths = sample_span_lengths(n_spans, lambda_param)
    
    # Sample starting positions for spans
    available_positions = list(range(n_tokens))
    span_starts = []
    
    for length in span_lengths:
        if not available_positions:
            break
        # Sample start position
        start = random.choice(available_positions)
        span_starts.append((start, length))
        
        # Remove positions covered by this span
        for i in range(start, min(start + length, n_tokens)):
            if i in available_positions:
                available_positions.remove(i)
    
    # Sort spans by position
    span_starts.sort()
    
    # Create corrupted input and target
    corrupted_input = []
    target_output = []
    sentinel_id = 0
    last_pos = 0
    
    for start, length in span_starts:
        # Add uncorrupted tokens before span
        corrupted_input.extend(tokens[last_pos:start])
        
        # Add sentinel token
        sentinel = f""
        corrupted_input.append(sentinel)
        
        # Add span to target with sentinel
        target_output.append(sentinel)
        end = min(start + length, n_tokens)
        target_output.extend(tokens[start:end])
        
        sentinel_id += 1
        last_pos = end
    
    # Add remaining uncorrupted tokens
    corrupted_input.extend(tokens[last_pos:])
    
    # Add final sentinel to target
    target_output.append(f"")
    
    return ' '.join(corrupted_input), ' '.join(target_output), span_starts

# Example
text = "The quick brown fox jumps over the lazy dog"
print(f"Original: {text}")
print(f"Tokens: {text.split()}")
print(f"Number of tokens: {len(text.split())}\n")

# Run span corruption
corrupted, target, spans = corrupt_with_spans(text, corruption_rate=0.15, lambda_param=3)

print(f"Corrupted input: {corrupted}")
print(f"Target output: {target}")
print(f"\nSpans corrupted: {spans}")

Original: The quick brown fox jumps over the lazy dog
Tokens: ['The', 'quick', 'brown', 'fox', 'jumps', 'over', 'the', 'lazy', 'dog']
Number of tokens: 9

Corrupted input: The quick  over  dog
Target output:  brown fox jumps  the lazy 

Spans corrupted: [(2, 3), (6, 2)]

from transformers import T5ForConditionalGeneration, T5Tokenizer
from transformers import Trainer, TrainingArguments
from datasets import load_dataset
import torch
from rouge_score import rouge_scorer

# Load T5-base model
model = T5ForConditionalGeneration.from_pretrained('t5-base')
tokenizer = T5Tokenizer.from_pretrained('t5-base')

# Load CNN/DailyMail dataset
dataset = load_dataset('cnn_dailymail', '3.0.0')

def preprocess_function(examples):
    """Format data as text-to-text"""
    # Add task prefix
    inputs = ["summarize: " + doc for doc in examples['article']]
    targets = examples['highlights']
    
    # Tokenize
    model_inputs = tokenizer(
        inputs, 
        max_length=512, 
        truncation=True,
        padding='max_length'
    )
    
    # Tokenize targets
    with tokenizer.as_target_tokenizer():
        labels = tokenizer(
            targets,
            max_length=128,
            truncation=True,
            padding='max_length'
        )
    
    model_inputs['labels'] = labels['input_ids']
    return model_inputs

# Preprocess dataset
tokenized_dataset = dataset.map(
    preprocess_function,
    batched=True,
    remove_columns=dataset['train'].column_names
)

# Training arguments
training_args = TrainingArguments(
    output_dir='./t5-summarization',
    num_train_epochs=3,
    per_device_train_batch_size=8,
    per_device_eval_batch_size=8,
    learning_rate=1e-4,
    warmup_steps=500,
    weight_decay=0.01,
    logging_dir='./logs',
    logging_steps=100,
    evaluation_strategy='epoch',
    save_strategy='epoch',
    load_best_model_at_end=True,
    fp16=True,  # Mixed precision training
)

# Create trainer
trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=tokenized_dataset['train'],
    eval_dataset=tokenized_dataset['validation'],
)

# Train model
trainer.train()

# Save model
model.save_pretrained('./t5-summarization-final')
tokenizer.save_pretrained('./t5-summarization-final')

def evaluate_rouge(model, tokenizer, test_dataset, num_samples=1000):
    """Evaluate model using ROUGE metrics"""
    scorer = rouge_scorer.RougeScorer(
        ['rouge1', 'rouge2', 'rougeL'], 
        use_stemmer=True
    )
    
    rouge_scores = {'rouge1': [], 'rouge2': [], 'rougeL': []}
    
    model.eval()
    for i in range(min(num_samples, len(test_dataset))):
        example = test_dataset[i]
        
        # Generate summary
        input_text = "summarize: " + example['article']
        input_ids = tokenizer(
            input_text, 
            return_tensors='pt',
            max_length=512,
            truncation=True
        ).input_ids
        
        with torch.no_grad():
            outputs = model.generate(
                input_ids,
                max_length=128,
                num_beams=4,
                length_penalty=0.6,
                early_stopping=True
            )
        
        generated_summary = tokenizer.decode(
            outputs[0], 
            skip_special_tokens=True
        )
        reference_summary = example['highlights']
        
        # Compute ROUGE scores
        scores = scorer.score(reference_summary, generated_summary)
        rouge_scores['rouge1'].append(scores['rouge1'].fmeasure)
        rouge_scores['rouge2'].append(scores['rouge2'].fmeasure)
        rouge_scores['rougeL'].append(scores['rougeL'].fmeasure)
    
    # Average scores
    avg_scores = {
        metric: sum(scores) / len(scores)
        for metric, scores in rouge_scores.items()
    }
    
    return avg_scores

# Evaluate T5
t5_scores = evaluate_rouge(model, tokenizer, dataset['test'])

print("T5-base ROUGE Scores:")
print(f"ROUGE-1: {t5_scores['rouge1']:.4f}")
print(f"ROUGE-2: {t5_scores['rouge2']:.4f}")
print(f"ROUGE-L: {t5_scores['rougeL']:.4f}")

from transformers import BartForConditionalGeneration, BartTokenizer

# Load BART-base
bart_model = BartForConditionalGeneration.from_pretrained('facebook/bart-base')
bart_tokenizer = BartTokenizer.from_pretrained('facebook/bart-base')

# Fine-tune BART (similar process, no task prefix needed)
# ... training code similar to T5 ...

# Evaluate BART
bart_scores = evaluate_rouge(bart_model, bart_tokenizer, dataset['test'])

print("\nBART-base ROUGE Scores:")
print(f"ROUGE-1: {bart_scores['rouge1']:.4f}")
print(f"ROUGE-2: {bart_scores['rouge2']:.4f}")
print(f"ROUGE-L: {bart_scores['rougeL']:.4f}")

from transformers import T5ForConditionalGeneration, T5Tokenizer
import torch

class T5TextToText:
    def __init__(self, model_name='t5-base'):
        self.model = T5ForConditionalGeneration.from_pretrained(model_name)
        self.tokenizer = T5Tokenizer.from_pretrained(model_name)
        self.model.eval()
    
    def predict(self, input_text, max_length=128):
        """Generate prediction for any text-to-text task"""
        input_ids = self.tokenizer(
            input_text,
            return_tensors='pt',
            max_length=512,
            truncation=True
        ).input_ids
        
        with torch.no_grad():
            outputs = self.model.generate(
                input_ids,
                max_length=max_length,
                num_beams=4,
                early_stopping=True
            )
        
        return self.tokenizer.decode(outputs[0], skip_special_tokens=True)

# Initialize model
t5 = T5TextToText()

# Part (a): Sentiment Classification
def sentiment_classification(text):
    """Convert sentiment classification to text-to-text"""
    input_text = f"sst2 sentence: {text}"
    prediction = t5.predict(input_text, max_length=10)
    return prediction

# Examples
examples_sentiment = [
    "This movie is absolutely fantastic!",
    "Terrible waste of time and money.",
    "It was okay, nothing special.",
]

print("=== Sentiment Classification ===")
for text in examples_sentiment:
    pred = sentiment_classification(text)
    print(f"Input: {text}")
    print(f"Prediction: {pred}\n")

# Part (b): Named Entity Recognition
def named_entity_recognition(text):
    """Convert NER to text-to-text"""
    # Format: extract entities from text
    input_text = f"ner: {text}"
    prediction = t5.predict(input_text, max_length=100)
    return prediction

# Alternative format: specific entity types
def ner_with_types(text, entity_type='person'):
    """Extract specific entity types"""
    input_text = f"extract {entity_type}: {text}"
    prediction = t5.predict(input_text, max_length=50)
    return prediction

# Examples
examples_ner = [
    "Apple Inc. was founded by Steve Jobs in Cupertino, California.",
    "Barack Obama was the 44th President of the United States.",
    "The Eiffel Tower is located in Paris, France.",
]

print("=== Named Entity Recognition ===")
for text in examples_ner:
    # General NER
    entities = named_entity_recognition(text)
    print(f"Input: {text}")
    print(f"Entities: {entities}")
    
    # Specific types
    persons = ner_with_types(text, 'person')
    locations = ner_with_types(text, 'location')
    organizations = ner_with_types(text, 'organization')
    
    print(f"Persons: {persons}")
    print(f"Locations: {locations}")
    print(f"Organizations: {organizations}\n")

# Part (c): Textual Entailment
def textual_entailment(premise, hypothesis):
    """Convert entailment to text-to-text"""
    input_text = f"mnli premise: {premise} hypothesis: {hypothesis}"
    prediction = t5.predict(input_text, max_length=20)
    return prediction

# Examples
examples_entailment = [
    {
        "premise": "A man is playing guitar on stage.",
        "hypothesis": "A person is performing music.",
        "label": "entailment"
    },
    {
        "premise": "A woman is reading a book in the library.",
        "hypothesis": "A woman is swimming in a pool.",
        "label": "contradiction"
    },
    {
        "premise": "The cat is sleeping on the couch.",
        "hypothesis": "The cat is dreaming.",
        "label": "neutral"
    },
]

print("=== Textual Entailment ===")
for ex in examples_entailment:
    pred = textual_entailment(ex['premise'], ex['hypothesis'])
    print(f"Premise: {ex['premise']}")
    print(f"Hypothesis: {ex['hypothesis']}")
    print(f"Prediction: {pred}")
    print(f"Ground truth: {ex['label']}\n")

=== Sentiment Classification ===
Input: This movie is absolutely fantastic!
Prediction: positive

Input: Terrible waste of time and money.
Prediction: negative

Input: It was okay, nothing special.
Prediction: neutral

=== Named Entity Recognition ===
Input: Apple Inc. was founded by Steve Jobs in Cupertino, California.
Entities: Apple Inc., Steve Jobs, Cupertino, California
Persons: Steve Jobs
Locations: Cupertino, California
Organizations: Apple Inc.

=== Textual Entailment ===
Premise: A man is playing guitar on stage.
Hypothesis: A person is performing music.
Prediction: entailment
Ground truth: entailment

Premise: A woman is reading a book in the library.
Hypothesis: A woman is swimming in a pool.
Prediction: contradiction
Ground truth: contradiction

Premise: The cat is sleeping on the couch.
Hypothesis: The cat is dreaming.
Prediction: neutral
Ground truth: neutral

training_examples = [
    # Sentiment
    {"input": "sst2 sentence: Great movie!", "target": "positive"},
    
    # NER
    {"input": "ner: John lives in NYC", "target": "John, NYC"},
    
    # Entailment
    {"input": "mnli premise: Cat sleeps hypothesis: Cat rests", 
     "target": "entailment"},
    
    # Translation
    {"input": "translate English to French: Hello", "target": "Bonjour"},
    
    # Summarization
    {"input": "summarize: [long article]", "target": "[summary]"},
]