← Roadmap

Learning Transferable Visual Models From Natural Language Supervision

Radford et al. · 2021

① Intuition

Before CLIP, the standard recipe for visual recognition was: collect a labelled dataset (ImageNet: 1.28M images, 1000 classes), train a supervised model, ship. This approach is powerful but brittle — if you want to classify a category not in those 1000 labels, you must re-label data and retrain. The label set is a hard ceiling.

CLIP (Contrastive Language-Image Pre-training) breaks that ceiling by learning from natural language supervision. Instead of fixed integer labels, each image is paired with a free-form caption scraped from the web. OpenAI assembled 400 million such (image, text) pairs. An image encoder and a text encoder are jointly trained so that matching pairs have high cosine similarity and mismatched pairs have low similarity. At inference time, zero-shot classification requires no additional training: you write text prompts (“a photo of a cat”, “a photo of a dog”, …) for any label set, encode them, and pick the label whose embedding is closest to the image embedding.

Before playing with the widget: if you lower the temperature in a softmax over similarity logits, what happens to the probability distribution? Does the diagonal get sharper or flatter?

Interact with the widget: select “image → text” and drag temperature to 0.10. Notice the diagonal turns dark blue and off-diagonal cells go nearly white — the model is very confident. Now raise temperature to 1.5: the matrix grays out and all cells look similar — the model is uncertain. This temperature parameter (tau in the paper) is learned during training, not hand-tuned.

CLIP Similarity Matrix

a photo of a cat
a photo of a dog
a photo of a car
a photo of a flower
🐱 cat
0.97
🐶 dog
0.97
🚗 car
0.99
🌸 flower
0.98

Switch to 'text → image' direction. Does the matrix change? Why or why not for a perfectly aligned square matrix?

Retrieval check

What is the key difference between CLIP's training signal and standard ImageNet supervised training?
CLIP achieves strong zero-shot ImageNet accuracy without being trained on ImageNet. Which property of its training makes zero-shot transfer possible?

Answer all 2 correctly to mark this stage mastered.

② Math

Given a training batch of N (image, text) pairs, CLIP computes two sets of embeddings:

I∈RN×d,T∈RN×d\mathbf{I} \in \mathbb{R}^{N \times d}, \quad \mathbf{T} \in \mathbb{R}^{N \times d}

Both are L2-normalized so that the dot product equals cosine similarity. The logit matrix is:

L=IT⊤τ\mathbf{L} = \frac{\mathbf{I} \mathbf{T}^\top}{\tau}

where tau is a learned (scalar) temperature parameter. The diagonal of L holds the scores for correct pairs; the off-diagonal entries are scores for incorrect pairs. The training loss is symmetric cross-entropy over the rows (image-to-text direction) and columns (text-to-image direction):

L=12[CE(L, y)+CE(L⊤, y)],y=[0,1,…,N−1]\mathcal{L} = \frac{1}{2}\left[\text{CE}(\mathbf{L},\, \mathbf{y}) + \text{CE}(\mathbf{L}^\top,\, \mathbf{y})\right], \quad \mathbf{y} = [0, 1, \ldots, N-1]

where CE(L, y) treats each row i as a softmax classification problem with correct class y_i = i, and CE(L^T, y) does the same column-wise.

Why does CLIP L2-normalize both I and T before computing the logit matrix?
Show next line
I^i=Ii∥Ii∥2⇒I^i⋅T^j=cos⁡θij∈[−1, 1]\hat{\mathbf{I}}_i = \frac{\mathbf{I}_i}{\|\mathbf{I}_i\|_2} \Rightarrow \hat{\mathbf{I}}_i \cdot \hat{\mathbf{T}}_j = \cos\theta_{ij} \in [-1,\,1]
Why does CLIP use symmetric cross-entropy — averaging the image-to-text loss and the text-to-image loss — rather than just one direction?
Show next line
L=12[CE(L,y)⏟image→text+CE(L⊤,y)⏟text→image]\mathcal{L} = \frac{1}{2}\bigl[\underbrace{\text{CE}(\mathbf{L},\mathbf{y})}_{\text{image}\to\text{text}} + \underbrace{\text{CE}(\mathbf{L}^\top,\mathbf{y})}_{\text{text}\to\text{image}}\bigr]

Retrieval check

In CLIP's loss, what is the role of the temperature parameter tau?
CLIP trains with batch size N = 32768. Why does a larger batch size improve the contrastive objective?

Answer all 2 correctly to mark this stage mastered.

③ Pseudo code

Arrange CLIP’s five-step contrastive training forward pass in the correct order:

  1. Encode all images in the batch: I_f = image_encoder(images) # (N, d)
  2. Encode all texts in the batch: T_f = text_encoder(texts) # (N, d)
  3. L2-normalize both: I = I_f / ||I_f||, T = T_f / ||T_f||
  4. Compute logit matrix: L = I @ T.T / temperature # (N, N)
  5. Compute symmetric CE loss: loss = (CE(L, labels) + CE(L.T, labels)) / 2

Fill in the blank in CLIP’s logit computation. I and T are already L2-normalized, temperature is a scalar:

def clip_logits(I, T, temperature):
    # I: (N, d), T: (N, d), both L2-normalized
    # returns: (N, N) logit matrix
    return ______

Retrieval check

In the pseudocode, why must L2-normalization happen before computing I @ T.T, not after?

Answer all 1 correctly to mark this stage mastered.

④ Code (Python)

Implement CLIP’s symmetric contrastive loss in NumPy. Given I (N x d) and T (N x d), both already L2-normalized, compute the scalar loss. Labels are the diagonal indices [0, 1, …, N-1].

Retrieval check

In the clip_loss implementation, why do we subtract l.max(axis=1, keepdims=True) before computing the log-sum-exp?

Answer all 1 correctly to mark this stage mastered.

⑤ Original paper

Your browser can't embed PDFs. Download CLIP: Learning Transferable Visual Models From Natural Language Supervision.

Open in new tab · Download

Suggested reading order: §2.3 (Contrastive Pre-training — the training objective and batch construction) → §3 (Experiments — breadth of zero-shot transfer) → §3.1.2 (Zero-Shot ImageNet — comparison to supervised baselines and few-shot ResNets).