Lensa ML
Lensa ML

How LSTMs Remember

Long Short-Term Memory networks explained

Step 1 of 6

Inside an LSTM Cell

The four components that give LSTMs their memory

C_t-1C_t×Forgetf_t = σ(...)+×i_t × c̃_tInputi_t = σ(...)Candidatec̃_t = tanh(...)Outputo_t = σ(...)tanh×h_t[ h_t-1 , x_t ]h_t-1x_tAll gates and candidate receive the same input — each learns different weights
GATES
3
FORGET
f_t
INPUT
i_t
OUTPUT
o_t

An LSTM cell has four key parts: a cell state that carries long-term memory, and three gates (forget, input, output) that control what to remember, write, and reveal. Click a button below to highlight each part.

Step 1 of 6: Inside an LSTM Cell

The four components that give LSTMs their memory

C_t-1C_t×Forgetf_t = σ(...)+×i_t × c̃_tInputi_t = σ(...)Candidatec̃_t = tanh(...)Outputo_t = σ(...)tanh×h_t[ h_t-1 , x_t ]h_t-1x_tAll gates and candidate receive the same input — each learns different weights
GATES
3
FORGET
f_t
INPUT
i_t
OUTPUT
o_t

An LSTM cell has four key parts: a cell state that carries long-term memory, and three gates (forget, input, output) that control what to remember, write, and reveal. Click a button below to highlight each part.

Step 2 of 6: Why LSTMs Work

How the cell state solves the vanishing gradient problem

Signal strength over time — RNN vs LSTMRNN80%t=056%t=139%t=227%t=319%t=4signalfades×0.7 per step (RNN) vs ~×1.0 per step (LSTM cell state)LSTM80%t=079%t=177%t=276%t=375%t=4signalstays
Weak (0.3)Strong (1.0)
Initial signal: 0.80
Short (3)Long (8)
Time steps: 5
RNN @ START
80%
RNN @ t=4
19%
LSTM @ START
80%
LSTM @ t=4
75%

Watch the circles: RNN signals shrink quickly (×0.7 each step) while LSTM signals stay nearly full (×0.97). The LSTM cell state acts like a highway — information flows straight through without repeated squashing. Try increasing the time steps to see the difference grow.

Step 3 of 6: The Forget Gate

Deciding what old information to throw away

Input: [ h_t-1, x_t ] = [0.12, -0.05, 0.30, 0.85, 0.10, -0.20]C_t-1 = [0.60, 0.68, 0.77]retained = [0.54, 0.27, 0.58]×Forget Gatef_t = σ([h_t-1, x_t])C_t-1 is a vector — each dim is multiplied by f_t independentlyC_t-1retained0.60×0.9→0.54dim 10.68×0.4→0.27dim 20.77×0.8→0.58dim 3
dim 1: forget (0)keep (1)
f_t[0] = 0.90
dim 2: forget (0)keep (1)
f_t[1] = 0.40
dim 3: forget (0)keep (1)
f_t[2] = 0.75
f_t[0]
0.90
f_t[1]
0.40
f_t[2]
0.75

The forget gate computes f_t = σ(W_f · [h_t-1, x_t] + b_f) — one value per dimension. Each slider controls one dimension of f_t. A 0 means "completely forget", while 1 means "keep entirely." Notice how different dims can forget different amounts.

Step 4 of 6: The Input Gate

Choosing what new information to store

Input: [ h_t-1, x_t ] = [0.12, -0.05, 0.30, 0.85, 0.10, -0.20]after forget = [0.54, 0.27, 0.58]+C_t = [0.93, 0.06, 0.82]Input Gatei_t = σ([h,x])Candidatec̃_t = tanh([h,x])×C_t = retained + (i_t × c̃_t) per dimensionretainedaddednew C_t0.54+0.39=0.93dim 10.27-0.21=0.06dim 20.58+0.24=0.82dim 3
dim 1: block (0)write (1)
i_t[0] = 0.70
dim 2: block (0)write (1)
i_t[1] = 0.85
dim 3: block (0)write (1)
i_t[2] = 0.30
dim 1: −1+1
c̃_t[0] = 0.55
dim 2: −1+1
c̃_t[1] = -0.25
dim 3: −1+1
c̃_t[2] = 0.80
C_t[0]
0.93
C_t[1]
0.06
C_t[2]
0.82

Both i_t = σ(W_i · [h_t-1, x_t] + b_i) and c̃_t = tanh(W_c · [h_t-1, x_t] + b_c) are computed from the same fixed input — but with different learned weights, so each dimension can write a different amount. Their product is added to the retained cell state.

Step 5 of 6: The Output Gate

Filtering the cell state into the hidden state

Input: [ h_t-1, x_t ] = [0.12, -0.05, 0.30, 0.85, 0.10, -0.20]C_t = [0.93, 0.06, 0.82]tanh[0.73, 0.06, 0.67]×Output Gateo_t = σ([h,x])h_t[0.58, 0.04, 0.61]h_t = o_t × tanh(C_t) per dimensiontanh(C_t)h_t0.73×0.800.58dim 10.06×0.600.04dim 20.67×0.900.61dim 3
dim 1: block (0)pass (1)
o_t[0] = 0.80
dim 2: block (0)pass (1)
o_t[1] = 0.60
dim 3: block (0)pass (1)
o_t[2] = 0.90
h_t[0]
0.58
h_t[1]
0.04
h_t[2]
0.61

o_t = σ(W_o · [h_t-1, x_t] + b_o) is a vector — each dimension independently controls how much of C_t to expose. C_t is passed through tanh (squashing to −1…1), then multiplied by o_t per dimension. The result is h_t — the hidden state sent to the next layer and the next time step.

Step 6 of 6: LSTM in Action

Processing a sentence word by word

Thecatsatonthe___
Unrolled LSTM — starting from C_t-1 = [0.60, 0.68, 0.77]LSTMTheLSTMcatLSTMsatLSTMonLSTMthesubjectverb-ctxposition
C_t[0]
1.10
C_t[1]
0.61
C_t[2]
0.63
h_t[0]
0.48
h_t[1]
0.27
h_t[2]
0.22

Starting from the same C_t-1 = [0.60, 0.68, 0.77] used in steps 3–5, each time step applies per-dimension gates (f_t, i_t, o_t) with different learned weights. Watch how subject info persists while verb-ctx and position evolve.