Transformer Architecture

The key insight of the Transformer is the self-attention mechanism, which allows the model to weigh the importance of different parts of the input when producing each output. When processing the word "bank" in a sentence, attention allows the model to look at surrounding context ("river" vs. "money") and dynamically determine the relevant meaning.

• • Multi-head attention: multiple attention heads run in parallel, each learning different relationship types.
• • Positional encoding: since Transformers process all tokens simultaneously, positional information is injected to preserve word order.
• • Encoder-Decoder: original Transformers used both; modern LLMs typically use decoder-only architectures.

Earlier models like LSTMs and GRUs processed tokens sequentially, making them slow to train and prone to forgetting information from early in long sequences. Transformers eliminated sequential processing entirely, enabling massive parallelization on GPUs and the ability to model long-range dependencies without the "vanishing gradient" problem.