← Roadmap

An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale

Dosovitskiy et al. · 2020

① Intuition

Before ViT, every competitive image model used convolutional neural networks. CNNs encode two strong inductive biases: locality (each filter sees only a small neighborhood) and translation equivariance (sliding a filter across the image). These biases are powerful — they let a ResNet generalize well even on modest datasets. But they also cap what the architecture can learn: a convolution can never directly relate a pixel in the top-left corner to one in the bottom-right.

ViT asks: what if we stripped away all CNN bias and applied a plain Transformer? The trick is treating the image as a sequence of patches. A 224×224 RGB image is cut into 196 non-overlapping 16×16 patches. Each patch is flattened to a 768-dimensional vector and linearly projected — exactly analogous to a word embedding. A learnable CLS token is prepended, learned position embeddings are added, and the resulting 197-token sequence is fed into a standard Transformer encoder. The CLS output is used for classification.

Interact with the widget: set Image size to 16 and Patch size to 4. Read off the number of tokens. Then switch Patch size to 8 — tokens drop by 4x. Observe how patch dim scales with patch area times channels.

Full image (16×16)
patches: 4
tokens per image: 16
patch dim: 4×4×3 = 48
Top-left patch (4× zoom)

ViT trained on ImageNet alone underperforms a ResNet of similar size. Trained on JFT-300M it matches or beats ResNets. Why does data scale compensate for missing CNN inductive bias?

Retrieval check

What is the purpose of the CLS token prepended to the patch sequence in ViT?
Why does ViT need substantially more training data than a CNN of comparable capacity to reach the same accuracy?

Answer all 2 correctly to mark this stage mastered.

② Math

The two key quantities in ViT are the sequence length N and the token dimension D:

N=H⋅WP2D=P2⋅CN = \frac{H \cdot W}{P^2} \qquad D = P^2 \cdot C

where H and W are image height and width, P is the patch size, and C is the number of channels. For the standard ViT-B/16 on 224x224 RGB images: N = 196, D = 768.

The linear patch embedding projects each flattened patch into D dimensions:

zi=xiWembed+bembed,xi∈RP2C,  Wembed∈RP2C×D\mathbf{z}_i = \mathbf{x}_i \mathbf{W}_{\text{embed}} + \mathbf{b}_{\text{embed}}, \quad \mathbf{x}_i \in \mathbb{R}^{P^2 C},\; \mathbf{W}_{\text{embed}} \in \mathbb{R}^{P^2 C \times D}

Self-attention then operates over the N+1 tokens (including CLS), with complexity:

O(N2⋅D)\mathcal{O}(N^2 \cdot D)

This is manageable at N=196 but grows rapidly with smaller patch sizes (e.g., P=8 gives N=784, roughly 16x more attention operations than P=16).

For a 224x224 RGB image with patch size P=16 and C=3 channels, compute N (sequence length) and D (token dimension).
Show next line
N=2242162=196,D=162×3=768N = \frac{224^2}{16^2} = 196, \quad D = 16^2 \times 3 = 768
Why does ViT need more data than a ResNet to generalize, from a Bayesian perspective?
Show next line
ResNet: tight spatial prior⇒less data neededViT: flat prior⇒more data needed\text{ResNet: tight spatial prior} \Rightarrow \text{less data needed} \quad \text{ViT: flat prior} \Rightarrow \text{more data needed}

Retrieval check

ViT-L/16 uses patch size 16. If you switched to ViT-L/8 (same image size 224x224), how does the sequence length change?
In the patch embedding equation z_i = x_i * W_embed, what is the shape of W_embed for ViT-B/16?

Answer all 2 correctly to mark this stage mastered.

③ Pseudo code

Arrange the five steps of ViT’s forward pass in the correct order:

  1. Add learned position embeddings to each of the N+1 token vectors
  2. Prepend a learnable CLS token to create a sequence of length N+1
  3. Feed the sequence through L Transformer encoder layers; use CLS output for classification
  4. Reshape image into N = (H*W)/P^2 non-overlapping patches of shape (P, P, C)
  5. Flatten each patch to a vector of dimension P^2*C and linearly project to D

Fill the blank in the patch embedding pseudocode. patch_embed has shape (N, P*P*C) and W_embed maps from P*P*C to D:

def embed_patches(patch_embed, W_embed, b_embed):
    # patch_embed: (N, P*P*C)
    # returns: (N, D)
    return ______

Retrieval check

In the ViT forward pass, why are position embeddings added after the linear patch projection rather than before?

Answer all 1 correctly to mark this stage mastered.

④ Code (Python)

Implement patchify in NumPy: given an image of shape (H, W, C) and patch size P, return a matrix of shape (N, P*P*C) where each row is a flattened patch. Patches are extracted in raster order (left-to-right, top-to-bottom).

Retrieval check

In the patchify implementation, why is transpose(0,2,1,3,4) needed after the first reshape?

Answer all 1 correctly to mark this stage mastered.

⑤ Original paper

Your browser can't embed PDFs. Download ViT: An Image is Worth 16x16 Words.

Open in new tab · Download

Suggested reading order: §3 (Method — full model description) → §3.1 (Patch Embedding — linear projection and position embeddings) → §4.2 (Scaling — JFT-300M results showing ViT beats ResNets at scale).