ENID
Lensa ML

How LSTMs Remember

Long Short-Term Memory networks explained

Step 1 of 6: Inside an LSTM Cell

The four components that give LSTMs their memory

C_t-1C_t×Forgetf_t=σ(…)+×i_t × c̃_tInputi_t=σ(…)Candidatec̃_t=tanh(…)Outputo_t=σ(…)tanh×h_t[ h_t-1 , x_t ]h_t-1x_tAll gates and candidate receive the same input; each learns different weights
Gates
3
Forget
f_t
Input
i_t
Output
o_t

An LSTM cell has four key parts: a cell state that carries long-term memory, and three gates (forget, input, output) that control what to remember, write, and reveal. Click a button below to highlight each part.

Click Cell State, Forget Gate, Input Gate, Output Gate, and Inputs in turn. The cell runs four small layers on the same input [h_t−1, x_t]: three sigmoid gates and one tanh candidate. With hidden size n and input size m, each layer has n × (n + m) weights plus n biases, so an LSTM has four times the parameters of a plain RNN of the same size. For n = m = 3, as here, that is 4 × (18 + 3) = 84. The 1997 original had no forget gate; Gers et al. added it in 2000.

Step 2 of 6: Why LSTMs Work

How the cell state solves the vanishing gradient problem

Signal strength over time — RNN vs LSTMRNN80%t=056%t=139%t=227%t=319%t=4signalfadesassumed: ×0.7 per step (RNN) vs ×f = 0.97 per step (LSTM cell state)LSTM80%t=078%t=175%t=273%t=371%t=4signalstays
Initial signal: 0.80
Weak (0.3)Strong (1.0)
Time steps: 5
Short (3)Long (8)
RNN @ start
80%
RNN @ t=4
19%
LSTM @ start
80%
LSTM @ t=4
71%

Watch the circles: RNN signals shrink quickly (×0.7 each step) while LSTM signals stay nearly full (×0.97, the forget gate value assumed here). On the cell state highway, information flows straight through without repeated squashing. Increase the time steps to see the gap grow.

Drag the Initial signal slider, then move Time steps from 3 to 8. The ×0.7 and ×0.97 rates are illustrative. The real reason lies in the derivative. Since C_t = f_t ⊙ C_(t−1) + i_t ⊙ c̃_t, the gradient of C_t with respect to C_(t−1) along this path is just f_t, with no weight matrix and no tanh derivative in between. Across k steps the gradient is a product of k forget gates, which stays near 1 while the gates stay open. Hochreiter and Schmidhuber called this path the constant error carousel.

Step 3 of 6: The Forget Gate

Deciding what old information to throw away

Input: [ h_t-1, x_t ] = [0.12, -0.05, 0.30, 0.85, 0.10, -0.20] · you set the gateC_t-1 = [0.60, 0.68, 0.77]retained = [0.54, 0.27, 0.58]×Forget gatef_t = σ([h_t-1, x_t])C_t-1 is a vector: each dim is multiplied by f_t independentlyC_t-1retained0.60×0.9→0.54dim 10.68×0.4→0.27dim 20.77×0.8→0.58dim 3
f_t[0] = 0.90
dim 1: forget (0)keep (1)
f_t[1] = 0.40
dim 2: forget (0)keep (1)
f_t[2] = 0.75
dim 3: forget (0)keep (1)
f_t[0]
0.90
f_t[1]
0.40
f_t[2]
0.75

The forget gate computes f_t = σ(W_f · [h_t-1, x_t] + b_f), one value per dimension. Each slider controls one dimension of f_t. A 0 means "completely forget", while 1 means "keep entirely." Notice how different dims can forget different amounts.

Drag the three f_t sliders and compare how much of C_t−1 = [0.60, 0.68, 0.77] each dimension keeps. A trained gate comes from a sigmoid, so it never reaches exactly 0 or 1. Frameworks often initialize the forget-gate bias b_f to 1, which starts f_t near σ(1) ≈ 0.73 so the cell remembers by default early in training. Jozefowicz et al. (2015) found that this one change closed most of the gap between the LSTM and other gated designs. Gates use σ rather than tanh because they act as multipliers between 0 and 1.

Step 4 of 6: The Input Gate

Choosing what new information to store

Input: [ h_t-1, x_t ] = [0.12, -0.05, 0.30, 0.85, 0.10, -0.20] · you set the gateafter forget = [0.54, 0.27, 0.58]+C_t = [0.93, 0.06, 0.82]Input gatei_t = σ([h,x])Candidatec̃_t=tanh([h,x])×C_t = retained + (i_t × c̃_t) per dimensionretainedaddednew C_t0.54+0.39=0.93dim 10.27-0.21=0.06dim 20.58+0.24=0.82dim 3
i_t[0] = 0.70
dim 1: block (0)write (1)
c̃_t[0] = 0.55
dim 1: −1+1
i_t[1] = 0.85
dim 2: block (0)write (1)
c̃_t[1] = -0.25
dim 2: −1+1
i_t[2] = 0.30
dim 3: block (0)write (1)
c̃_t[2] = 0.80
dim 3: −1+1
C_t[0]
0.93
C_t[1]
0.06
C_t[2]
0.82

Both i_t = σ(W_i · [h_t-1, x_t] + b_i) and c̃_t = tanh(W_c · [h_t-1, x_t] + b_c) are computed from the same fixed input but with different learned weights, so each dimension can write a different amount. Their product is added to the retained cell state.

Drag the i_t and c̃_t sliders and watch the three C_t readouts. With the starting values, dim 1 becomes 0.90 × 0.60 + 0.70 × 0.55 = 0.925. The candidate uses tanh so it can push memory down as well as up: dim 2's candidate of −0.25 lowers that cell to about 0.06. The update is a sum, and that addition is what lets gradients flow back without shrinking. Some variants tie the gates so that i_t = 1 − f_t, and the cell writes only as much as it forgets. The GRU uses this coupling.

Step 5 of 6: The Output Gate

Filtering the cell state into the hidden state

Input: [ h_t-1, x_t ] = [0.12, -0.05, 0.30, 0.85, 0.10, -0.20] · you set the gateC_t = [0.93, 0.06, 0.82]tanh[0.73, 0.06, 0.67]×Output gateo_t = σ([h,x])h_t[0.58, 0.04, 0.61]h_t = o_t × tanh(C_t) per dimensiontanh(C_t)h_t0.73×0.800.58dim 10.06×0.600.04dim 20.67×0.900.61dim 3
o_t[0] = 0.80
dim 1: block (0)pass (1)
o_t[1] = 0.60
dim 2: block (0)pass (1)
o_t[2] = 0.90
dim 3: block (0)pass (1)
h_t[0]
0.58
h_t[1]
0.04
h_t[2]
0.61

o_t = σ(W_o · [h_t-1, x_t] + b_o) is a vector: each dimension controls how much of C_t to expose. C_t is passed through tanh (squashing to −1…1), then multiplied by o_t per dimension. The result is h_t, the hidden state sent to the next layer and the next time step.

Drag the three o_t sliders and watch h_t change while C_t stays fixed. This split means an LSTM carries two states: C_t is internal memory, and h_t is what the next layer and the next time step see. With the default gates, h_t[0] = 0.80 × tanh(0.925) ≈ 0.58. A closed output gate can hold a stored fact for many steps and reveal it only when needed, such as keeping track of whether a subject is singular until the verb that must agree with it. Stacked LSTMs feed h_t upward as the next layer's x_t.

Step 6 of 6: LSTM in Action

Processing a sentence word by word

Thecatsatonthe___
Unrolled LSTM — starting from C_t-1 = [0.60, 0.68, 0.77]LSTMTheLSTMcatLSTMsatLSTMonLSTMthesubjectverb-ctxposition
t = 0Time step 0: "The"t = 4
C_t[0]
1.10
C_t[1]
0.61
C_t[2]
0.63
h_t[0]
0.48
h_t[1]
0.27
h_t[2]
0.22

Starting from the same C_t-1 = [0.60, 0.68, 0.77] used in steps 3–5, each time step applies per-dimension gates (f_t, i_t, o_t). The gate values are hand-set per word to stand in for learned weights, and "mat" is the target word, not a decoded output. Watch how subject info persists while verb-ctx and position evolve.

Press Pause, then click the word buttons to inspect each time step of "The cat sat on the". The gate values here are hand-written to show the idea; a trained model computes them from its weights at every step. The final h_t would feed a softmax layer over the vocabulary to predict the next word, whose target in this demo is "mat". Google's 2016 neural translation system was built from stacked LSTMs, and the sequence-to-sequence model of Sutskever et al. (2014) used one LSTM to read the input and another to write the output.

What is an LSTM (long short-term memory) network?

An LSTM (long short-term memory) network is a recurrent neural network with a separate memory, the cell state, and three gates that decide what to forget, what to write, and what to output at each step. The gates are small sigmoid layers that output values between 0 and 1, and they multiply the information flowing through the cell. Because the cell state is updated by addition, an LSTM can keep information for hundreds of steps where a plain RNN loses it after a few.

LSTMs were the standard sequence model from the late 2000s until Transformers arrived. They were used in speech recognition, machine translation, handwriting recognition, and text prediction, and they are still used for time series forecasting, sensor data, and small on-device models. Understanding the cell state also makes residual connections and gated layers in newer architectures easier to follow.

Key ideas

  • An LSTM adds a cell state that is updated by addition instead of being rewritten at each step.
  • The forget gate scales old memory, the input gate scales new content, and the output gate scales what is exposed.
  • Each gate is a sigmoid layer applied to [h_t−1, x_t], with one value per dimension.
  • The cell update is C_t = f_t ⊙ C_(t−1) + i_t ⊙ c̃_t, and the output is h_t = o_t ⊙ tanh(C_t).
  • Along the cell state, the gradient is multiplied only by forget gates, so it can survive long sequences.
  • An LSTM has four times the weights of a plain RNN with the same hidden size.

Common questions

What is the difference between an LSTM and a GRU?

A GRU (gated recurrent unit) merges the cell state and hidden state into one vector and uses two gates, update and reset, instead of three. It has about three quarters of the parameters of an LSTM of the same size. In comparisons the two usually perform similarly, so the choice is often made by speed or by trying both.

Are LSTMs better than Transformers?

For most language tasks with plenty of data, Transformers are more accurate and train faster because they process all positions in parallel. LSTMs still do well on small datasets, streaming input, and devices with little memory, since each new step costs a fixed amount of compute. For time series both are used.

What do input size and hidden size mean in an LSTM?

Input size is the length of each x_t, for example the dimension of a word embedding or the number of sensor readings per step. Hidden size is the length of h_t and C_t, which sets how much the cell can remember. Libraries such as PyTorch take both as arguments, along with the number of stacked layers.

What is a bidirectional LSTM?

A bidirectional LSTM runs one LSTM from left to right and another from right to left, then concatenates their hidden states at each position. Each word's representation then reflects context on both sides. It is used for tagging and classification, but not for generating text, where the future is not yet known.

Related explainers

References

  1. Hochreiter & Schmidhuber (1997), "Long Short-Term Memory", Neural Computation
  2. Gers, Schmidhuber & Cummins (2000), "Learning to Forget: Continual Prediction with LSTM", Neural Computation
  3. Greff et al. (2017), "LSTM: A Search Space Odyssey"
  4. Cho et al. (2014), "Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation"
  5. Dive into Deep Learning, "Long Short-Term Memory (LSTM)"

Last updated