Vision Transformers (ViT): How an Image Becomes a Sequence of Patches

The Vision Transformer, usually shortened to ViT, is the model that showed a standard Transformer - the architecture behind large language models - could classify images without the convolutional layers computer vision had relied on for most of the previous decade. It was introduced in the paper An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale, submitted to arXiv on 22 October 2020 by Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov and colleagues at Google Research, and published at ICLR 2021.

From pixels to tokens

A language Transformer reads a sequence of tokens. ViT makes an image look like one:

1. Split the image into patches. The image is cut into a grid of fixed-size squares. In the ViT-B/16 configuration each patch is 16 x 16 pixels, which is where the paper's title comes from. A 224 x 224 image becomes a 14 x 14 grid - 196 patches.
2. Flatten and project. Each patch is flattened into a vector and passed through a learned linear projection, producing one embedding per patch - the image equivalent of a word embedding.
3. Add position. A learned position embedding is added to each patch embedding, because the Transformer itself has no built-in notion of where a patch sits in the image.
4. Add a class token. A learnable extra token is prepended to the sequence. After the Transformer encoder runs, the output at that position is passed to a classification head.

Everything after step 4 is a standard Transformer encoder: layers of multi-head self-attention and feed-forward blocks, as described in Attention Is All You Need.

What it gave up, and what that cost

Convolutional networks build in two assumptions about images: that nearby pixels are related (locality) and that a pattern means the same thing wherever it appears (translation equivariance). The ViT paper describes these as inductive biases that a Transformer lacks.

The consequence reported in the paper is that data volume decides the outcome. Trained on mid-sized datasets such as ImageNet without strong regularisation, ViT models landed a few percentage points below comparable ResNets. Pretrained on larger collections - ImageNet-21k, with about 14 million images, and Google's internal JFT-300M, with about 300 million - they matched or exceeded state-of-the-art convolutional networks when transferred to benchmarks including ImageNet, CIFAR-100 and the VTAB suite, while, in the authors' words, "requiring substantially fewer computational resources to train".

The model sizes

The paper defined three sizes, borrowing names from BERT:

Variant	Layers	Hidden size	Heads	Parameters
ViT-Base	12	768	12	86M
ViT-Large	24	1024	16	307M
ViT-Huge	32	1280	16	632M

The suffix names the patch size: ViT-L/16 uses 16 x 16 patches and ViT-H/14 uses 14 x 14. Smaller patches mean more tokens per image and more compute, because self-attention cost grows with the square of sequence length.

Where ViT shows up now

The patch-embedding idea outlived the original classifier. Image encoders built on ViT sit inside many later systems - the image encoder in Meta's Segment Anything is a ViT pretrained with masked autoencoding, and the vision side of many multimodal language models also starts by turning an image into patch tokens. That is why vendors of models with image input, listed with their specifications on the models page, describe image cost in tokens rather than pixels - see how tokens are counted.

For the wider field this sits in, see Computer Vision Explained.