Attention Is All You Need: The 2017 Paper Behind the Transformer

Almost every model on this site's models page is built on the Transformer, the architecture introduced in Attention Is All You Need. The paper was submitted to arXiv on 12 June 2017 and presented at NeurIPS 2017. Its eight authors - Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser and Illia Polosukhin - were at Google Brain, Google Research and the University of Toronto.

The problem it addressed

In 2017 the leading models for machine translation were recurrent neural networks (RNNs), which read a sentence one word at a time, each step depending on the one before. That sequential dependency limited how much of the work could run in parallel on GPUs, and made it harder to connect words far apart in a sentence.

The paper's proposal was to drop recurrence and convolutions entirely and build the model from attention alone.

Self-attention

Attention lets every position in a sequence look at every other position directly and decide how much each one matters. For each token, the model computes three vectors from its embedding:

• a query - what this token is looking for,
• a key - what this token offers to others,
• a value - the information it passes on.

The paper's scaled dot-product attention compares each query against all keys, scales the scores by the square root of the key dimension, turns them into weights with a softmax, and uses those weights to mix the values:

Attention(Q, K, V) = softmax(QK^T / sqrt(d_k)) V

Because every token's attention is computed at once as matrix multiplication, the whole sequence can be processed in parallel.

Multi-head attention and positions

Instead of a single attention calculation, the model runs several in parallel - heads - each with its own learned projections, so different heads can track different kinds of relationships. Their outputs are concatenated and projected back.

Attention on its own has no sense of word order. The paper adds positional encodings - sine and cosine functions of different frequencies - to the input embeddings so the model can use each token's position.

The architecture

The model is an encoder-decoder. The encoder reads the source sentence; the decoder generates the translation one token at a time, attending both to its own output so far and to the encoder's output. Each layer pairs attention with a feed-forward network, with residual connections and layer normalisation around each.

The base model's dimensions:

Setting	Base model
Encoder layers	6
Decoder layers	6
Model dimension	512
Attention heads	8
Feed-forward dimension	2048

The results reported

On the WMT 2014 English-to-German translation task the paper reported 28.4 BLEU, ahead of earlier published results including ensembles. On WMT 2014 English-to-French it reported 41.8 BLEU for a single model, after training for 3.5 days on eight GPUs - a fraction of the training cost of the best earlier models, according to the authors.

What came after

Later families kept the attention mechanism and changed the layout. BERT (2018) used only the encoder stack. The GPT series used only the decoder stack, generating text left to right - the layout most of today's chat and coding models descend from, as described in How Large Language Models Work. The Vision Transformer applied the same encoder to image patches.

The cost of attention grows with the square of the sequence length, because every token attends to every other. Much later research - and the engineering behind today's long context windows - is about reducing that cost.

For more foundational papers, see Foundational AI Research Papers Worth Reading and Scaling Laws and Chinchilla.