① Intuition
Before BERT, the dominant language model paradigm was unidirectional: GPT reads tokens strictly left-to-right (causal masking), meaning when predicting or encoding position i, the model only sees tokens to the left. This is natural for generation, but it’s a significant limitation for understanding — knowing both the left and right context of a word is crucial for tasks like question answering and named entity recognition.
BERT’s key insight: instead of predicting the next token, mask out random tokens and predict them using the full surrounding context — bidirectional attention in both directions simultaneously. This is called Masked Language Modeling (MLM). On top of that, BERT adds Next Sentence Prediction (NSP): given two sentences A and B, predict whether B is the actual next sentence in the corpus or a random one. Together, these two pretraining objectives allow BERT to build rich, context-aware representations of text.
Why can't GPT-style (left-to-right) pretraining learn bidirectional context, even if you stack many layers?
Causal masking is applied at the attention layer: each token’s attention is literally blocked from attending to any future position. No amount of depth overcomes this architectural constraint — the information never enters the computation graph. BERT removes causal masking entirely during pretraining, letting every token attend to every other token.
Interact with the widget below to see how MLM masking works versus the causal (GPT-style) masking pattern. Toggle between modes and observe which positions can attend to which.
| [CLS] | The | cat | sat | on | mat | |
|---|---|---|---|---|---|---|
| [CLS] | ||||||
| The | ||||||
| cat | ||||||
| sat | ||||||
| on | ||||||
| mat |
After playing with the widget: BERT replaces only 80% of the selected masked tokens with [MASK]. The other 10% get a random token, and 10% stay unchanged. Why not replace all 100% with [MASK]?
At fine-tuning time, the input never contains literal [MASK] tokens — [MASK] is a pretraining-only artifact. If the model only ever sees [MASK] tokens in the positions it needs to predict, it learns to activate its prediction machinery only when [MASK] is present, causing a train-test mismatch. The 10% random and 10% unchanged tokens force the model to maintain a useful representation for every token, since any token could be the one it needs to predict.
Retrieval check
Answer all 2 correctly to mark this stage mastered.
② Math
BERT’s pretraining loss is the sum of two cross-entropy terms. First, MLM loss sums over the set M of masked positions:
Here the conditioning context is the full input with masked positions replaced — the model predicts the original token x_i using all non-masked context. Second, NSP loss uses the [CLS] token’s pooled representation:
The total pretraining loss is simply their sum:
Show next line
Show next line
Retrieval check
Answer all 2 correctly to mark this stage mastered.
③ Pseudo code
Arrange the BERT pretraining step in the correct order:
- Sample a batch of sentence pairs (A, B) from the corpus
- Randomly select 15% of token positions; replace with [MASK] / random / unchanged per 80/10/10 rule
- Forward pass through all Transformer layers to get contextual hidden states
- Backpropagate total loss L = L_MLM + L_NSP and update all parameters
- Compute L_MLM = -Σ log p(x_i | x_{∖M}) over masked positions + L_NSP from [CLS]
Fill the blank in the masking pseudocode:
for i in selected_positions:
r = random()
if r < 0.80:
tokens[i] = ______
elif r < 0.90:
tokens[i] = random_vocab_token()
# else: leave unchangedRetrieval check
Answer all 1 correctly to mark this stage mastered.
④ Code (Python)
Implement the BERT 15% masking procedure in NumPy. Given an array of token IDs and a mask_token_id, return a new array where approximately 15% of positions are selected. Of those selected positions: 80% become mask_token_id, 10% become a random token from the vocabulary, and 10% stay unchanged. Also return a boolean array indicating which positions were selected (for computing the MLM loss).
Retrieval check
Answer all 1 correctly to mark this stage mastered.
⑤ Original paper
Suggested reading order: §3.1 (Model Architecture — the Transformer encoder stack and input representations) → §3.2 (Pre-training BERT — MLM and NSP objectives with the 80/10/10 masking scheme) → §5 (Ablation Studies — shows the contribution of each pretraining task and the bidirectionality).