① Intuition
Before ViT, every competitive image model used convolutional neural networks. CNNs encode two strong inductive biases: locality (each filter sees only a small neighborhood) and translation equivariance (sliding a filter across the image). These biases are powerful — they let a ResNet generalize well even on modest datasets. But they also cap what the architecture can learn: a convolution can never directly relate a pixel in the top-left corner to one in the bottom-right.
ViT asks: what if we stripped away all CNN bias and applied a plain Transformer? The trick is treating the image as a sequence of patches. A 224×224 RGB image is cut into 196 non-overlapping 16×16 patches. Each patch is flattened to a 768-dimensional vector and linearly projected — exactly analogous to a word embedding. A learnable CLS token is prepended, learned position embeddings are added, and the resulting 197-token sequence is fed into a standard Transformer encoder. The CLS output is used for classification.
Interact with the widget: set Image size to 16 and Patch size to 4. Read off the number of tokens. Then switch Patch size to 8 — tokens drop by 4x. Observe how patch dim scales with patch area times channels.
ViT trained on ImageNet alone underperforms a ResNet of similar size. Trained on JFT-300M it matches or beats ResNets. Why does data scale compensate for missing CNN inductive bias?
CNN locality and translation equivariance are implicit priors that help generalization when data is scarce. With millions of images, these priors become unnecessary — a Transformer has enough examples to learn the spatial structure of images from scratch. Large-scale pretraining lets ViT discover patch relationships that CNNs hard-code, and even go beyond them (e.g., attending across the entire image from layer 1).
Retrieval check
Answer all 2 correctly to mark this stage mastered.
② Math
The two key quantities in ViT are the sequence length N and the token dimension D:
where H and W are image height and width, P is the patch size, and C is the number of channels. For the standard ViT-B/16 on 224x224 RGB images: N = 196, D = 768.
The linear patch embedding projects each flattened patch into D dimensions:
Self-attention then operates over the N+1 tokens (including CLS), with complexity:
This is manageable at N=196 but grows rapidly with smaller patch sizes (e.g., P=8 gives N=784, roughly 16x more attention operations than P=16).
Show next line
Show next line
Retrieval check
Answer all 2 correctly to mark this stage mastered.
③ Pseudo code
Arrange the five steps of ViT’s forward pass in the correct order:
- Add learned position embeddings to each of the N+1 token vectors
- Prepend a learnable CLS token to create a sequence of length N+1
- Feed the sequence through L Transformer encoder layers; use CLS output for classification
- Reshape image into N = (H*W)/P^2 non-overlapping patches of shape (P, P, C)
- Flatten each patch to a vector of dimension P^2*C and linearly project to D
Fill the blank in the patch embedding pseudocode. patch_embed has shape (N, P*P*C) and W_embed maps from P*P*C to D:
def embed_patches(patch_embed, W_embed, b_embed):
# patch_embed: (N, P*P*C)
# returns: (N, D)
return ______Retrieval check
Answer all 1 correctly to mark this stage mastered.
④ Code (Python)
Implement patchify in NumPy: given an image of shape (H, W, C) and patch size P, return a matrix of shape (N, P*P*C) where each row is a flattened patch. Patches are extracted in raster order (left-to-right, top-to-bottom).
Retrieval check
Answer all 1 correctly to mark this stage mastered.
⑤ Original paper
Suggested reading order: §3 (Method — full model description) → §3.1 (Patch Embedding — linear projection and position embeddings) → §4.2 (Scaling — JFT-300M results showing ViT beats ResNets at scale).