① Intuition
Before CLIP, the standard recipe for visual recognition was: collect a labelled dataset (ImageNet: 1.28M images, 1000 classes), train a supervised model, ship. This approach is powerful but brittle — if you want to classify a category not in those 1000 labels, you must re-label data and retrain. The label set is a hard ceiling.
CLIP (Contrastive Language-Image Pre-training) breaks that ceiling by learning from natural language supervision. Instead of fixed integer labels, each image is paired with a free-form caption scraped from the web. OpenAI assembled 400 million such (image, text) pairs. An image encoder and a text encoder are jointly trained so that matching pairs have high cosine similarity and mismatched pairs have low similarity. At inference time, zero-shot classification requires no additional training: you write text prompts (“a photo of a cat”, “a photo of a dog”, …) for any label set, encode them, and pick the label whose embedding is closest to the image embedding.
Before playing with the widget: if you lower the temperature in a softmax over similarity logits, what happens to the probability distribution? Does the diagonal get sharper or flatter?
Lowering temperature divides the logits by a smaller number, making the differences between them larger before the softmax. A larger spread of inputs to softmax produces a sharper (more peaked) distribution. So at low temperature the diagonal entries — the correct image-text matches — dominate strongly, and off-diagonal entries collapse toward zero.
Interact with the widget: select “image → text” and drag temperature to 0.10. Notice the diagonal turns dark blue and off-diagonal cells go nearly white — the model is very confident. Now raise temperature to 1.5: the matrix grays out and all cells look similar — the model is uncertain. This temperature parameter (tau in the paper) is learned during training, not hand-tuned.
CLIP Similarity Matrix
Switch to 'text → image' direction. Does the matrix change? Why or why not for a perfectly aligned square matrix?
For a square matrix with the same four items on both axes, the visual pattern stays the same — the diagonal is still the correct match in both directions. But the probabilities differ: row-softmax normalizes each image’s scores over all texts, while column-softmax normalizes each text’s scores over all images. CLIP’s training objective explicitly pushes both directions simultaneously, which is why the loss is symmetric.
Retrieval check
Answer all 2 correctly to mark this stage mastered.
② Math
Given a training batch of N (image, text) pairs, CLIP computes two sets of embeddings:
Both are L2-normalized so that the dot product equals cosine similarity. The logit matrix is:
where tau is a learned (scalar) temperature parameter. The diagonal of L holds the scores for correct pairs; the off-diagonal entries are scores for incorrect pairs. The training loss is symmetric cross-entropy over the rows (image-to-text direction) and columns (text-to-image direction):
where CE(L, y) treats each row i as a softmax classification problem with correct class y_i = i, and CE(L^T, y) does the same column-wise.
Show next line
Show next line
Retrieval check
Answer all 2 correctly to mark this stage mastered.
③ Pseudo code
Arrange CLIP’s five-step contrastive training forward pass in the correct order:
- Encode all images in the batch: I_f = image_encoder(images) # (N, d)
- Encode all texts in the batch: T_f = text_encoder(texts) # (N, d)
- L2-normalize both: I = I_f / ||I_f||, T = T_f / ||T_f||
- Compute logit matrix: L = I @ T.T / temperature # (N, N)
- Compute symmetric CE loss: loss = (CE(L, labels) + CE(L.T, labels)) / 2
Fill in the blank in CLIP’s logit computation. I and T are already L2-normalized, temperature is a scalar:
def clip_logits(I, T, temperature):
# I: (N, d), T: (N, d), both L2-normalized
# returns: (N, N) logit matrix
return ______Retrieval check
Answer all 1 correctly to mark this stage mastered.
④ Code (Python)
Implement CLIP’s symmetric contrastive loss in NumPy. Given I (N x d) and T (N x d), both already L2-normalized, compute the scalar loss. Labels are the diagonal indices [0, 1, …, N-1].
Retrieval check
Answer all 1 correctly to mark this stage mastered.
⑤ Original paper
Suggested reading order: §2.3 (Contrastive Pre-training — the training objective and batch construction) → §3 (Experiments — breadth of zero-shot transfer) → §3.1.2 (Zero-Shot ImageNet — comparison to supervised baselines and few-shot ResNets).