1Text Classification & Sentiment 2Seq2Seq & Beam Search 3Named Entity Recognition 4Language Modeling 5BERT & Masked Language Modeling

Lensa ML

Language Modeling

Step 1 of 6: What Is a Language Model?

Predicting P(next token | context) over a vocabulary

Position in sentence2

Context

the cat sat

P(next)

74.9%

Given context "the cat sat", the model predicts P(next token). Here the actual next word "on" gets 74.9%, and every other vocabulary word still gets some share. (This toy distribution is hand-set so the actual word leads. A real model spreads probability over every plausible continuation, and the word that actually comes next is often not its top pick.)

Drag Position in sentence from 0 to 4 to change how much of "the cat sat on the mat" the model has seen. By the chain rule, next-token predictions also score whole sentences: P(the cat sat) = P(the) · P(cat | the) · P(sat | the cat). Before neural networks, n-gram models estimated these terms from counts and kept only the last n − 1 words of context. Bengio et al. (2003) replaced the counts with a neural network over word embeddings, and Transformers later stretched the context to thousands of tokens.

Step 2 of 6: Autoregressive Generation

Each predicted token feeds back as input for the next

Generation step0

Step

Context

the

Predicts

big

Generation starts from the seed token "the". With only that one token as context, the model predicts "big".

Drag Generation step from 0 to 6 to grow "the big cat sat on a mat" one token at a time. Each forward pass produces only one new token, so a 500-token answer takes 500 passes, which is why chatbot replies stream out word by word. Models avoid recomputing the whole prefix at every step by caching the attention keys and values of earlier tokens, a store called the KV cache. Training needs no loop: a causal mask lets the model predict every position of a sentence in one parallel pass.

Step 3 of 6: Training with Cross-Entropy

The loss that teaches models to assign high probability to correct tokens

Model confidence on correct token0.70

Confidence

70%

CE loss (nats)

0.357

Entropy (nats)

1.270

Target

sat

Moderate confidence (70%): loss = 0.357. The loss still charges for the probability not given to "sat", and training pushes it higher.

Drag Model confidence on correct token from 0.05 to 0.95 and watch the loss. With a one-hot target, cross-entropy reduces to −log p of the correct token: p = 0.05 costs about 3.0 nats, p = 0.5 costs 0.69, and p = 0.95 costs 0.05. A model that guesses uniformly over this 10-word vocabulary pays ln 10 ≈ 2.30 per token. Pretraining minimizes this single loss averaged over trillions of tokens, and Kaplan et al. (2020) found it falling smoothly as a power law in model size, data, and compute.

Step 4 of 6: Temperature & Sampling

Control the sharpness of the probability distribution

Temperature1.00

Temperature

1.00

Entropy (nats)

1.442

Top-1 prob

50.9%

Diversity

Medium

T = 1.00: moderate temperature. "the" is favored, but alternatives keep real probability. At T=1.0 the distribution is exactly what the model learned.

Drag Temperature from 0.1 to 2.0 and watch the bars sharpen and flatten. The logits are divided by T before the softmax, p_i = exp(z_i / T) / Σ_j exp(z_j / T), so the ranking of tokens never changes, only the gaps between them. As T approaches 0, sampling turns into greedy decoding, and as T grows it approaches a uniform pick. Temperature acts only at generation time. The trained weights, and the loss the model reached, stay exactly the same.

Step 5 of 6: Top-k Sampling

Filter low-probability tokens to prevent nonsense

Top-k3

Top-k

Mass kept

86.9%

Filtered

Top-3: only the 3 most probable tokens are kept (86.9% of total mass). Crossed-out tokens can never be sampled, which blocks unlikely words but leaves some variety.

Drag Top-k from 1 to 10 and watch the kept tokens get renormalized. A fixed k has a weakness: when the model is sure, k = 40 still admits junk, and when many words fit, k = 3 cuts good ones. Nucleus (top-p) sampling, from Holtzman et al. (2019), keeps the smallest set of tokens whose probabilities sum to at least p, often 0.9, so the set grows and shrinks with the model's uncertainty. The same paper showed that greedy and beam decoding make text loop and repeat, which is one reason chat models sample.

Step 6 of 6: Perplexity

Measuring how 'surprised' the model is by the data

Model quality0.60

Perplexity

2.7

Cross-entropy (nats)

0.995

Quality

60%

Eff. vocab

Perplexity 2.7: a decent model. Function words ("the", "on") still get high probability, while content words ("cat", "mat") are less certain and add most of the loss.

Drag Model quality from 0.1 to 0.95 and watch perplexity fall. Perplexity is e raised to the average cross-entropy, PPL = exp(−(1/N) Σ log p_i), which equals the geometric mean of 1/p over the tokens. A uniform guess over V tokens scores exactly V. Perplexity only compares models that share a tokenizer, because splitting text into more, easier tokens lowers it. On the WikiText-103 benchmark, the largest GPT-2 model reached a perplexity of about 17.5 without training on that dataset.

What is a language model?

A language model is a model that assigns a probability to every possible next token given the tokens before it. Multiplying those next-token probabilities scores a whole sentence, and sampling from them one token at a time generates new text. It is trained by reading large amounts of text and minimizing cross-entropy on each next token.

Language models power autocomplete, speech recognition, machine translation, and chat assistants such as ChatGPT and Claude, which are large Transformer language models fine-tuned to follow instructions. Because predicting the next word well requires grammar, facts, and some reasoning, pretraining on this one objective produces models that can be adapted to summarization, coding, question answering, and many other tasks.

Key ideas

A language model outputs a probability distribution over the next token given the previous ones.
The chain rule turns next-token probabilities into the probability of a whole sequence.
Generation is autoregressive: each sampled token is appended to the context.
Training minimizes cross-entropy, which is −log p of the correct next token.
Temperature, top-k, and top-p change how tokens are sampled, not what the model learned.
Perplexity, the exponential of average cross-entropy, measures how well a model predicts held-out text.

Common questions

What is the difference between a language model and a large language model?

Nothing in the objective. A large language model (LLM) is a language model, almost always a decoder-only Transformer, with billions of parameters trained on trillions of tokens. Chat assistants add further training on instructions and human feedback on top of that base model.

What is the difference between causal and masked language modeling?

A causal language model predicts each token from the tokens before it, which is what GPT-style models do and what makes them able to generate text. A masked language model, like BERT, hides some tokens and predicts them from context on both sides. Masked models are better encoders for understanding tasks but cannot generate text left to right.

Why do language models make things up?

The training objective rewards text that is probable given the context, not text that is true. When the model lacks the fact it needs, a fluent and confident wrong continuation can still score as likely. Retrieval, tool use, and training that rewards saying "I don't know" reduce the problem but do not remove it.

What is bits per character?

It is cross-entropy measured in bits (log base 2) and divided by the number of characters instead of tokens. Because it does not depend on the tokenizer, it allows fair comparison between models that split text differently, which per-token perplexity does not.

Related explainers

Tokenization & BPE — How text becomes numbers — from characters to subword tokens via byte-pair encoding
Softmax, Logits & Temperature — Turning raw scores into probabilities
Entropy & Cross-Entropy — Measuring surprise, uncertainty, and prediction quality
Decoder-Only Transformers — The architecture behind GPT and modern LLMs
The Transformer Architecture — A visual guide to 'Attention Is All You Need'
BERT & Masked Language Modeling — Mask a word, predict it from both sides — how BERT learns deep bidirectional representations

References

Last updated October 25, 2026