Lensa ML
Lensa ML

Language Modeling

Step 1 of 6

What Is a Language Model?

Predicting P(next token | context) over a vocabulary

Position in sentence2
thecatsat→?P(next | context):4.6%the4.8%cat4.4%sat74.9%on2.7%mat2.0%dog1.6%ran1.6%big1.8%a1.6%is
CONTEXT
the cat sat
NEXT
on
P(NEXT)
74.9%

Given context "the cat sat", the model predicts P(next token). The correct next word "on" has the highest probability (74.9%). A language model assigns probabilities to every possible continuation.

Step 1 of 6: What Is a Language Model?

Predicting P(next token | context) over a vocabulary

Position in sentence2
thecatsat→?P(next | context):4.6%the4.8%cat4.4%sat74.9%on2.7%mat2.0%dog1.6%ran1.6%big1.8%a1.6%is
CONTEXT
the cat sat
NEXT
on
P(NEXT)
74.9%

Given context "the cat sat", the model predicts P(next token). The correct next word "on" has the highest probability (74.9%). A language model assigns probabilities to every possible continuation.

Step 2 of 6: Autoregressive Generation

Each predicted token feeds back as input for the next

Generation step0
thefull context (1 token)Modelbig→ feeds back as context1/7 tokens
STEP
1
CONTEXT
the
PREDICTS
big

Generation starts with a seed token "the". The model takes this single token as context and predicts the next most likely token "big".

Step 3 of 6: Training with Cross-Entropy

The loss that teaches models to assign high probability to correct tokens

Model confidence on correct token0.70
Target (one-hot):thecatsatonmatdogranbigaisModel prediction:3.3%the3.3%cat70.0%sat3.3%on3.3%mat3.3%dog3.3%ran3.3%big3.3%a3.3%isCross-Entropy Loss = 0.357
CONFIDENCE
70%
CE LOSS
0.357
ENTROPY
1.832
TARGET
sat

Moderate confidence (70%) — loss = 0.357. Cross-entropy penalizes the model for not being confident enough about the correct token "sat". Training pushes this probability higher.

Step 4 of 6: Temperature & Sampling

Control the sharpness of the probability distribution

Temperature1.00
softmax(logits / T) — T = 1.0050.9%the20.7%cat15.3%sat4.2%on3.4%mat2.1%dog1.5%ran0.9%big0.6%a0.3%is
TEMPERATURE
1.00
ENTROPY
2.081
TOP-1 %
50.9%
DIVERSITY
Medium

T = 1.00 — moderate temperature. The top token "the" is favored but alternatives have nonzero probability. This balances quality and diversity. T=1.0 is the model's "natural" distribution.

Step 5 of 6: Top-k Sampling

Filter low-probability tokens to prevent nonsense

Top-k3
Top-3 sampling — keep 3 most likely tokens58.6%the23.8%cat17.6%satonmatdogranbigaisCumulative mass:86.9%
TOP-K
3
MASS KEPT
86.9%
FILTERED
7

Top-3: only the 3 most probable tokens are kept (86.9% of total mass). Crossed-out tokens are impossible to sample. This prevents unlikely tokens from appearing while allowing some diversity.

Step 6 of 6: Perplexity

Measuring how 'surprised' the model is by the data

Model quality0.60
the54.2%0.61cat25.2%1.38sat33.9%1.08on51.3%0.67the55.4%0.59mat19.4%1.64P−log pavg −log p = 0.995 → PPL = e^{avg} = e^{0.995}Perplexity = 2.71
PERPLEXITY
2.7
CROSS-ENTROPY
0.995
QUALITY
60%
EFF. VOCAB
~3

Perplexity 2.7 — decent model. Notice how function words ("the", "on") still get high probability, but content words ("cat", "mat") are less certain. PPL = e^(average −log p) — it's the geometric mean of 1/p across all tokens.