← Roadmap

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Devlin, Chang, Lee, Toutanova · 2018

① Intuition

Before BERT, the dominant language model paradigm was unidirectional: GPT reads tokens strictly left-to-right (causal masking), meaning when predicting or encoding position i, the model only sees tokens to the left. This is natural for generation, but it’s a significant limitation for understanding — knowing both the left and right context of a word is crucial for tasks like question answering and named entity recognition.

BERT’s key insight: instead of predicting the next token, mask out random tokens and predict them using the full surrounding context — bidirectional attention in both directions simultaneously. This is called Masked Language Modeling (MLM). On top of that, BERT adds Next Sentence Prediction (NSP): given two sentences A and B, predict whether B is the actual next sentence in the corpus or a random one. Together, these two pretraining objectives allow BERT to build rich, context-aware representations of text.

Why can't GPT-style (left-to-right) pretraining learn bidirectional context, even if you stack many layers?

Interact with the widget below to see how MLM masking works versus the causal (GPT-style) masking pattern. Toggle between modes and observe which positions can attend to which.

[CLS]Thecatsatonmat
[CLS]
The
cat
sat
on
mat
Attendable Masked (causal)Active row = thick blue border (query position 0: [CLS])

After playing with the widget: BERT replaces only 80% of the selected masked tokens with [MASK]. The other 10% get a random token, and 10% stay unchanged. Why not replace all 100% with [MASK]?

Retrieval check

What architectural change allows BERT to use bidirectional context?
What are the two pretraining objectives in the original BERT paper?

Answer all 2 correctly to mark this stage mastered.

② Math

BERT’s pretraining loss is the sum of two cross-entropy terms. First, MLM loss sums over the set M of masked positions:

LMLM=−∑i∈Mlog⁡p(xi∣x∖M)L_\text{MLM} = -\sum_{i \in M} \log p(x_i \mid x_{\setminus M})

Here the conditioning context is the full input with masked positions replaced — the model predicts the original token x_i using all non-masked context. Second, NSP loss uses the [CLS] token’s pooled representation:

LNSP=−log⁡p(IsNext∣[CLS])L_\text{NSP} = -\log p(\text{IsNext} \mid [\text{CLS}])

The total pretraining loss is simply their sum:

L=LMLM+LNSPL = L_\text{MLM} + L_\text{NSP}
BERT masks 15% of tokens, not 50%. If you masked half the tokens, the model would see less context per prediction. What is the fundamental signal/noise tradeoff that pins the masking rate near 15%?
Show next line
mask rate≈15%: balance context (noise ↓) vs. signal density\text{mask rate} \approx 15\%: \text{ balance context (noise } \downarrow\text{) vs. signal density}
Of the 15% selected positions, BERT replaces 80% with [MASK], 10% with a random token, and 10% unchanged. Why are the 10% random and 10% unchanged cases necessary?
Show next line
80% [MASK] + 10% random + 10% unchanged⇒no train-test [MASK] mismatch\text{80\% [MASK] + 10\% random + 10\% unchanged} \Rightarrow \text{no train-test [MASK] mismatch}

Retrieval check

In the MLM loss, x_{∖M} denotes the input with masked positions replaced. What token fills those masked positions in the input to the model?
Which token's representation is used for the NSP classification head?

Answer all 2 correctly to mark this stage mastered.

③ Pseudo code

Arrange the BERT pretraining step in the correct order:

  1. Sample a batch of sentence pairs (A, B) from the corpus
  2. Randomly select 15% of token positions; replace with [MASK] / random / unchanged per 80/10/10 rule
  3. Forward pass through all Transformer layers to get contextual hidden states
  4. Backpropagate total loss L = L_MLM + L_NSP and update all parameters
  5. Compute L_MLM = -Σ log p(x_i | x_{∖M}) over masked positions + L_NSP from [CLS]

Fill the blank in the masking pseudocode:

for i in selected_positions:
    r = random()
    if r < 0.80:
        tokens[i] = ______
    elif r < 0.90:
        tokens[i] = random_vocab_token()
    # else: leave unchanged

Retrieval check

In the BERT training loop, the masking decision (which 15% of positions to mask) is made once per sequence before the forward pass. Why is the mask re-sampled each epoch rather than fixed across training?

Answer all 1 correctly to mark this stage mastered.

④ Code (Python)

Implement the BERT 15% masking procedure in NumPy. Given an array of token IDs and a mask_token_id, return a new array where approximately 15% of positions are selected. Of those selected positions: 80% become mask_token_id, 10% become a random token from the vocabulary, and 10% stay unchanged. Also return a boolean array indicating which positions were selected (for computing the MLM loss).

Retrieval check

In the masking implementation, why do we use rng.random() < 0.80 for [MASK] and rng.random() < 0.90 for random token (instead of e.g. separate 0.80 and 0.10 comparisons)?

Answer all 1 correctly to mark this stage mastered.

⑤ Original paper

Your browser can't embed PDFs. Download BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.

Open in new tab · Download

Suggested reading order: §3.1 (Model Architecture — the Transformer encoder stack and input representations) → §3.2 (Pre-training BERT — MLM and NSP objectives with the 80/10/10 masking scheme) → §5 (Ablation Studies — shows the contribution of each pretraining task and the bidirectionality).