ENID
Lensa ML

BERT & Masked LM

Step 1 of 6: Bidirectional Context

Looking left AND right

Focus token3
Bidirectional: attend to BOTH left and right context0.19The0.15cat0.24saton0.15the0.16warm0.11mat← Left-to-right (GPT)"The cat sat __"Can only see: The cat sat↔ Bidirectional (BERT)"The cat sat [MASK] the warm mat"Sees: The cat sat the warm matBERT sees the full sentence (minus the masked token) from both sides.
Focus
"on"
Left context
3
Right context
3

"on" has 3 tokens of left context (pink) and 3 tokens of right context (teal). BERT attends to both sides at once, which a left-to-right model cannot do. (The weights shown are illustrative; they sum to 1 across the other tokens.)

Drag Focus token from 0 to 6 to move through "The cat sat on the warm mat" and compare its left and right context. A bidirectional model cannot be trained by next-word prediction: if every layer sees every position, the word to predict is already in the input. BERT's answer is to hide words and predict them, a task the paper names after the cloze test from 1950s reading research. BERT-base has 12 encoder layers, 768 hidden units, and 110 million parameters; BERT-large has 24 layers and 340 million.

Step 2 of 6: Masking Strategy

Choosing which tokens to hide

Mask probability (%)15
Original:Thecatsatonthewarmmat↓ mask 15%Input to BERT:Thecatsatonthe[MASK]matBERT's masking strategy (of selected tokens):80% → [MASK] tokenForces model to predict from context10% → Random wordAny visible word may be a swap: check them all10% → Keep originalTeaches model to verify correct words1 of 7 tokens selected (14%) — BERT only trains on predicting these
Mask rate
15%
[MASK]
1
Random
0
Kept
0

15% masking: roughly 1 in 7 tokens are selected. The original BERT paper uses 15%: enough targets to learn from, with most of the input intact. Why not always use [MASK]? Fine-tuning inputs never contain [MASK]. Leaving 20% of the selected tokens as real words (10% swapped, 10% unchanged) means the model can't tell which visible words it will be graded on, so it learns a good representation of every token, not only of [MASK] positions.

Drag Mask probability (%) from 0 to 50 and watch which tokens are picked. Of the selected tokens, BERT replaces 80% with [MASK], 10% with a random word, and leaves 10% unchanged, so at a 15% selection rate only about 12% of tokens show [MASK]. Later work revisited the rate: Wettig et al. (2022) found that masking 40% matched or beat 15% for large models. SpanBERT masks contiguous spans of words, and RoBERTa draws a new mask each time it sees a sentence (dynamic masking).

Step 3 of 6: Predicting [MASK]

Filling in the blank

Mask position1
Strategy: [MASK] token (80%)0
80% of selected tokens → replaced with [MASK]BERT sees:The[MASK]satonthewarmmatTarget: always predict original "cat"Top-5 predictions (true: "cat"):cat91.2%✓ truedog4.1%the1.1%hat0.7%cold0.7%+ 5 more words sharing 2.3% probability
Strategy
[MASK] token
True token
"cat"
Top prediction
cat
True rank
#1

[MASK] strategy (80% of cases): BERT sees "[MASK]" and correctly predicts "cat" with 91.2% confidence, using only the surrounding words.

Drag Mask position from 0 to 6, then move the Strategy slider through [MASK] token, Random word, and Keep original. The prediction comes from BERT's final vector at that position, passed through a small output layer that shares weights with the input embedding matrix, then a softmax over the 30,522-token WordPiece vocabulary. The loss counts only the selected positions, so each sentence trains on about 15% of its tokens. GPT-style training gets a target at every position, and ELECTRA later recovered that efficiency for encoders.

Step 4 of 6: Pre-training Loss

Cross-entropy on masked tokens

Epoch0
LossEpoch1350102030Predictions across all 3 masking strategies:80% [MASK][MASK] → ?cat: 14%dog: 86%10% Random"ran" → ?sat: 8%ran: 92%10% Keep"warm" → ?warm: 13%cold: 87%
Epoch
0
Loss
5.42
Avg P(true token)
12%

Epoch 0: before training, predictions are random for all three strategies and the loss is high. In each box the target is the true token. Press ▶ Play or drag the slider. (The loss curve and probabilities in this step are illustrative, not from a real training run.)

Click ▶ Play or drag the Epoch slider from 0 to 30 to watch the loss fall. The real run used BooksCorpus (800 million words) and English Wikipedia (2.5 billion words) for 1 million steps with batches of 256 sequences, about 40 passes over the data. BERT also trained a second objective, next sentence prediction (NSP), which asks whether sentence B really follows sentence A in the source text. RoBERTa later showed that dropping NSP and training longer on more data gave better results.

Step 5 of 6: Fine-tuning

From pre-trained to task-specific

Fine-tuning epoch0
Pre-trained BERT + classification head[CLS]greatmovie[SEP]BERT (frozen or fine-tuned)h_[CLS]Linear + SoftmaxP(positive | review) = 50%Confusion matrixPred +Pred −True +2624True −2426Fine-tuning: add a small classification head on top of [CLS] embeddingTrain for just 2-4 epochs on labeled data — pre-trained knowledge transfers
Fine-tune epoch
0
Accuracy
51.2%
True positives
26
False negatives
24

Before fine-tuning: the [CLS] embedding carries general language knowledge, but the new classifier head is untrained, so accuracy sits near 50% (chance). (The accuracy curve, confusion matrix and example probability are illustrative, not from a real fine-tuning run.)

Drag Fine-tuning epoch from 0 to 20 and watch accuracy and the confusion matrix. For classification, the final vector of the [CLS] token feeds one linear layer, and all of BERT's weights are updated along with it. The paper recommends 2 to 4 epochs, batch sizes of 16 or 32, and learning rates between 2e-5 and 5e-5, much smaller than in pre-training. Question answering uses the same encoder with a different head that predicts the start and end of the answer span in the passage.

Step 6 of 6: Contextual Embeddings

Same word, different meanings

Polysemous word0
"bank" in different contexts → different embeddingsWord2Vec (static)financial institutionriverbankdepositedriverapprovedsatExample contexts:"I deposited money at the bank" → financial institution"The river bank was muddy" → riverbank"The bank approved the loan" → financial institution"We sat on the bank fishing" → riverbank
Word
"bank"
Meanings
2
Contexts
4
Type
Contextual

"bank" has 2 distinct meanings: financial institution vs. riverbank. Word2Vec maps all uses to one point (amber circle). BERT gives each use its own embedding: the "financial institution" uses cluster together (teal), and the "riverbank" uses form a separate cluster (pink).

Drag Polysemous word from 0 to 2 to compare bank, bat, and light across four sentences each. Every vector depends on the whole sentence, and different layers hold different information: Tenney et al. (2019) found that lower layers capture more syntax and higher layers more meaning. Whole-sentence vectors are a separate problem. Averaging raw BERT token vectors works poorly for similarity search, which is why Sentence-BERT fine-tunes BERT on sentence pairs to produce vectors that can be compared with cosine similarity.

What is BERT and masked language modeling?

BERT (Bidirectional Encoder Representations from Transformers) is a Transformer encoder pre-trained to fill in hidden words using context from both sides. This training task is called masked language modeling: about 15% of the tokens in each sentence are hidden, and the model learns to predict them. After pre-training, BERT produces a contextual vector for every token and can be fine-tuned for a specific task with one small extra layer.

Google released BERT in 2018, and it set new records on 11 language-understanding benchmarks at once. It made the pre-train then fine-tune recipe standard in NLP. BERT-style encoders still run search ranking, text classification, named entity recognition, extractive question answering, and the embedding models behind semantic search and retrieval-augmented generation.

Key ideas

  • BERT is a Transformer encoder that reads the whole sentence in both directions at once.
  • Masked language modeling hides about 15% of tokens and trains the model to predict them.
  • Selected tokens become [MASK] 80% of the time, a random word 10%, and stay unchanged 10%.
  • The loss is cross-entropy on the selected positions only.
  • Fine-tuning adds a small task head and updates all weights for a few epochs.
  • Each occurrence of a word gets its own vector, so different senses separate.

Common questions

What is the difference between BERT and GPT?

Both are built from Transformer blocks. BERT is an encoder with attention in both directions, trained to fill in masked words, which makes it strong at understanding tasks. GPT is a decoder with causal attention, trained to predict the next word, which lets it generate text.

Can BERT generate text?

Not well. It predicts masked tokens in parallel from a fixed-length input and has no built-in left-to-right process for writing new text. Generation needs a decoder, as in GPT, or an encoder-decoder model such as T5 or BART.

What are the [CLS] and [SEP] tokens?

[CLS] is added at the start of every input, and its final vector acts as a summary of the whole sequence for classification. [SEP] marks the boundary between two segments, such as a question and a passage, and BERT also adds a segment embedding so each token knows which part it belongs to.

Is BERT still used?

Yes. Large generative models handle many tasks through prompting, but BERT-style encoders are far cheaper to run and remain the usual choice for high-volume classification, ranking, and embeddings. Newer encoders such as RoBERTa, DeBERTa, and ModernBERT follow the same masked-language-model design.

Related explainers

References

  1. Devlin et al. (2019), "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding", NAACL
  2. Liu et al. (2019), "RoBERTa: A Robustly Optimized BERT Pretraining Approach"
  3. Clark et al. (2020), "ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators", ICLR
  4. Wettig et al. (2022), "Should You Mask 15% in Masked Language Modeling?"
  5. Reimers & Gurevych (2019), "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks"

Last updated