BERT & Masked LM

"on" has 3 tokens of left context (pink) and 3 tokens of right context (teal). BERT attends to BOTH simultaneously — this is why it outperforms left-to-right models on understanding tasks.

15% masking: roughly 1 in 7 tokens are selected. BERT's original paper uses 15% — enough signal to learn, but not so much that the input is unrecognizable.

[MASK] strategy (80% of cases): BERT sees "[MASK]" and correctly predicts "cat" with 91.2% confidence. This is the primary training signal — predict the hidden word from context.

Epoch 0: before training, the model predicts randomly across all three strategies. Loss is high (~4.5). The [MASK], random-word, and keep-original boxes all train BERT to predict the true token. Press ▶ Play or drag the slider.

Before fine-tuning: the [CLS] embedding has general language knowledge but no task-specific signal. Accuracy starts around 50% (random). The pre-trained BERT weights already encode syntax and semantics.

"bank" has 2 distinct meanings: financial institution vs. riverbank. Word2Vec maps all uses to one point (amber circle). BERT creates different embeddings for each context — the "financial institution" uses cluster together (teal) and the "riverbank" uses cluster separately (pink). This is the power of contextual embeddings!