How Transformers Work

TL;DR — Key Takeaways

• The Decoder generates output sequentially, one token per step, feeding each result back as input for the next.
• Masked Self-Attention (Causal Masking) prevents the model from “cheating” by looking at future tokens it hasn’t generated yet.
• Cross-Attention is the bridge: the Decoder’s queries meet the Encoder’s keys and values to pull in relevant input context.
• A Linear + Softmax head converts the Decoder’s output vector into a probability over the entire vocabulary.
• Three Transformer variants exist: Encoder-only (BERT), Decoder-only (GPT), and Encoder-Decoder (T5).

 

Step 8: How the Decoder Works

The Decoder’s job is to generate the output sequence, one token at a time. Given the input question “What is the capital of France?”, the Encoder has already built a rich understanding of every word. The Decoder now uses that understanding to produce an answer.
Each Decoder layer contains three sub-layers, applied in order: Masked Self-Attention, Cross-Attention, and a Feed-Forward Network — each wrapped in residual connections and layer norm, just like the Encoder.

The process begins with a special start token <START>. From there, the Decoder predicts the next word, appends it, and repeats — until it generates the <END> token.

 

Step 9a: Masked Self-Attention (Causal Masking)

In the Encoder, every token attends to every other token freely. The Decoder cannot do this: while generating text, future tokens don’t exist yet. If the model could see them, it would simply copy the answer rather than learning to generate it.

In the Encoder, every token attends to every other token freely. The Decoder cannot do this: while generating text, future tokens don’t exist yet. If the model could see them, it would simply copy the answer rather than learning to generate it.

Example: The Decoder has generated <START>, “Par”] and is now predicting “is.” In masked self-attention, “is” can look at <START> and “Par” but the attention scores for future positions are set to negative infinity — which Softmax converts to zero.

This restriction is called causal masking because it enforces the causal structure of generation: the past causes the present, the present causes the future.

 

Step 9b: Cross-Attention — The Bridge

Cross-attention is what makes the Encoder’s understanding accessible to the Decoder. After masked self-attention processes the Decoder’s own previous tokens, cross-attention queries the Encoder’s output for relevant input information.

Worked Example: Generating “Paris”

The Encoder has read “What is the capital of France?” and built contextual representations for every word. The Decoder starts with <START>.

In cross-attention, the Decoder asks: “What do I need to generate next?” (this becomes Q). The Encoder provides Keys and Values for all six input words: What, is, the, capital, of, France.

The Decoder’s query compares against every encoder key. “capital” and “France” score highest because they contain the information most relevant to answering the question. Their corresponding Values are weighted most heavily. The resulting cross-attention output tells the Decoder: “The answer is a city related to France.”

This is called cross-attention because it crosses the boundary between the two components: Q from the Decoder, K and V from the Encoder. Contrast this with self-attention where all three come from the same sequence.

 

Step 10: Feed-Forward Network in the Decoder

After masked self-attention and cross-attention have done their work, the decoder representation passes through the same Feed-Forward Network structure used in the Encoder: two linear layers with ReLU activation, applied independently to each token.

If cross-attention says “focus on France and capital,” the FFN sharpens that signal — strengthening the features that lead to predicting “Paris” and suppressing weaker or noisy signals. Each token is refined independently before the final prediction step.

 

Picking the Next Word: Linear + Softmax

After all three sub-layers process a token’s representation, the result is still just a vector of numbers — not a word. Two final layers convert that vector into a concrete word selection.

The Linear Layer

A linear projection expands the decoder’s output vector into a score for every word in the vocabulary. For a vocabulary of 50,000 words, this produces 50,000 raw scores called logits. Higher logit = higher confidence.

Softmax

Softmax converts raw logits into proper probabilities that sum to 1.0. The highest probability word is selected as the next generated token.

“Paris” wins with 0.91 probability. The output becomes [“<START>”, “Paris”]. “Paris” is then fed back into the Decoder to predict the next token — and this loop repeats until <END> is generated.

 

The Complete Data Flow

Here is the full journey from input text to generated output, combining both Part 1 and Part 2.

Encoder Side (Steps 1–7)

• Input text tokenized into tokens
• Tokens → Token IDs (vocabulary lookup)
• Token IDs → Embedding vectors (embedding matrix)
• Positional encoding added
• N × [Multi-Head Self-Attention + Add & Norm + Feed-Forward + Add & Norm]
• Contextual representations passed to Decoder

Decoder Side (Steps 8–10)

• Start with <START> token
• N × [Masked Self-Attention + Cross-Attention + Feed-Forward], all with Add & Norm
• Linear layer → vocabulary scores
• Softmax → probabilities
• Highest probability word selected as next token
• New token appended and fed back → repeat until <END>

 

The Three Transformer Variants

Not every application needs both an Encoder and a Decoder. Modern AI systems use three configurations depending on the task.

Note: GPT-style models (Decoder-only) do not use a separate Encoder. The Decoder processes the prompt and generates continuation directly, using causal masking throughout.

 

Why the Transformer Is So Powerful

 

Component Quick Reference

— End of Part 3 — You now understand how Transformers work, end to end.