1Convolutional Neural Networks 2Recurrent Neural Networks 3LSTMs 4Autoencoders & VAEs 5GANs 6Skip Connections & ResNets

Lensa ML

How RNNs Read Sequences

Explained through sentiment analysis

Step 1 of 6: The Task: Read & Judge

Given a movie review, decide — positive or negative?

Rearrange the same words

Themoviewasnotgood

Word order

OriginalSwappedShuffled

Sentiment

Negative

Score

-0.81

Words → Embeddings → RNN

The↓[0.1,0.0,
0.0,0.0]

→

movie↓[0.1,0.3,
0.1,0.1]

→

was↓[0.0,0.0,
0.0,0.0]

…

Each word is mapped to a numeric vector (embedding) before the RNN can process it

Word order matters

"not good" ≠ "good not"

Words → vectors

Embeddings encode meaning

Sequence needed

Can't judge one word alone

Drag the slider to rearrange the words. Word order matters: "not good" is negative, while "good not" makes little sense. Each word becomes a numeric embedding vector, and the RNN reads these one at a time, left to right. The score comes from the small RNN built in the next steps. It rates all three orders negative, because its embedding for "not" is strongly negative on its own; the order only changes how negative.

Drag the WORD ORDER slider from Original to Swapped to Shuffled. A bag-of-words model counts the same five words in all three orderings and cannot tell them apart. An RNN processes the embeddings in order, so position changes the result. The embeddings here have 4 numbers per word and are hand-picked for the demo. Real models learn them, usually with 100 to 1,000 dimensions. Text is first split into tokens, and each token ID selects a row of an embedding table that is trained with the rest of the network.

Step 2 of 6: Reading Word by Word

An RNN reads left to right, updating its understanding at each step

Reading:(start)

Time step: 0 / 7

Initial Hidden State

h₀=[+0.000,+0.000,+0.000,+0.000]

All zeros: the RNN has read nothing yet. Step forward to see the math.

h0[0]

0.000

h0[1]

0.000

h0[2]

0.000

h0[3]

0.000

Sentiment

0.000

Step

0/7

Step through to see the full RNN calculation with real numbers. At each step: multiply the previous hidden state by W_h, the word embedding by W_x, add them, and apply tanh. Click the element buttons to expand each dot product. Watch how "not" flips the hidden state negative.

Drag the TIME STEP slider or use Prev and Next to read the review one word at a time. The hidden state here has 4 numbers, and W_h is 0.98 on its diagonal and 0 elsewhere, so each step keeps 98% of every value before the new word is added. Real RNNs learn a full W_h and add a bias: h_t = tanh(W_h · h_(t−1) + W_x · x_t + b). Tanh keeps every value between −1 and +1, which stops the state from growing without bound on long sequences.

Step 3 of 6: The Hidden State Evolves

Watch the network's "mood" shift as it reads each word

Themoviewasnotreallythatgood

Step through: 0 / 7

Weight Sharing

Every time step uses the same W_h and W_x: the RNN cell is reused, not copied.

Unrolled, the RNN is one cell repeated at each step. All copies share identical weights, so the parameter count does not grow with sequence length. The sentiment bar shows how the internal state evolves as each word is read.

Drag the STEP THROUGH slider and watch the bars across the 7 words of "The movie was not really that good". Unrolling is also how RNNs are trained. The unrolled chain is treated as one deep network and gradients flow back through every copy, which is called backpropagation through time (BPTT). For long texts the chain is often cut into fixed-length chunks (truncated BPTT) to save memory. Because each step needs the one before it, an RNN cannot process a sequence in parallel, so training is slower than for a Transformer.

Step 4 of 6: Long Reviews, Short Memory

Vanilla RNNs forget earlier words — a critical flaw

Sequence depth: 6 steps

2 (short)8 (long)

First step grad

0.0503

Last step grad

1.0000

Ratio

5.0%

Increase the depth to see gradients vanish exponentially. Each backward step multiplies the gradient by W_h·tanh′. Here we assume that factor is 0.55, so after 6 steps the first word's gradient is 0.0503. Early words barely affect learning. This is the vanishing gradient problem. (Our toy RNN uses W_h = 0.98, so over its 7 words the gradient only shrinks to about 0.3–0.85; real RNNs with smaller weights or longer inputs decay like this chart.)

Drag SEQUENCE DEPTH from 2 to 8. The decay factor of 0.55 is fixed for the demo. In a real RNN the gradient passing through each step is multiplied by W_h and by the tanh derivative, which is at most 1. When those factors are below 1 the gradient shrinks exponentially, and when they are above 1 it explodes. Exploding gradients have a simple fix: gradient clipping caps the gradient's norm. Vanishing gradients need a different design. Bengio, Simard and Frasconi described the problem in 1994.

Step 5 of 6: LSTM: Selective Memory

Gates let the network choose what to remember and forget

Forget Gate: 0.50

Input Gate: 0.50

Output Gate: 0.50

Prev cell

0.80

New cell

0.700

Hidden out

0.302

f × c_prev

0.400

i × cand

0.300

o × tanh(c)

0.302

Drag each gate slider to see its effect. Forget gate at 0 erases old memory; at 1 it keeps everything. Input gate controls how much new info is written. Output gate filters what gets passed to the next step. Formula: c_t = f·c_(t-1) + i·candidate, h_t = o·tanh(c_t).

Drag the Forget, Input, and Output sliders. The demo fixes the old cell value at 0.8 and the candidate at 0.6, so with all three gates at 0.5 the new cell is 0.5 × 0.8 + 0.5 × 0.6 = 0.7. In a trained LSTM nobody sets the gates by hand. Each is a sigmoid of the current input and previous hidden state. The cell update is additive, which lets gradients pass back through many steps while the forget gate stays near 1. The GRU (2014) is a lighter variant with two gates.

Step 6 of 6: The Final Verdict

The last hidden state holds enough context to classify the whole review

Time step: 0 / 7

h[0]

0.000

h[1]

0.000

h[2]

0.000

h[3]

0.000

Sentiment

0.000

Prediction

—

Strength |score|

0.00

NegativePositive

Step through each word to watch the RNN build its verdict. After reading the entire sequence, the final hidden state goes through an output layer to give a score. Watch how "not" before "good" makes the RNN output Negative.

Click Negative review and Positive review, then step through each word. In the negative review the score drops to about −0.98 at "not" and ends at −0.81, so "good" pulls it back only part of the way. Only the final hidden state is used here, a many-to-one setup. Other tasks use the states differently. A tagger reads out a label at every step (many-to-many), and a bidirectional RNN adds a second pass from right to left so each word's state also sees what comes after it.

What is a recurrent neural network (RNN)?

A recurrent neural network (RNN) is a neural network that reads a sequence one element at a time and carries a hidden state from each step to the next. At every step it combines the previous hidden state with the current input using the same weights, so the hidden state acts as a running summary of everything read so far. The final state, or the state at each step, feeds an output layer.

RNNs were the main model for text, speech, and time series before Transformers. They powered early neural machine translation, speech recognition, handwriting recognition, and next-word prediction on phone keyboards. They still matter for streaming data and small devices because each new input costs a fixed amount of compute, and their training problems explain why LSTMs, GRUs, and attention were invented.

Key ideas

An RNN updates a hidden state with h_t = tanh(W_h · h_(t−1) + W_x · x_t + b).
The same weights are reused at every time step.
The hidden state is a fixed-size summary of everything read so far.
Training unrolls the network and runs backpropagation through time.
Gradients shrink or grow exponentially with sequence length.
Gated cells such as the LSTM and GRU were designed to keep long-range information.

Common questions

What is the difference between an RNN and a feedforward network?

A feedforward network maps one fixed-size input to one output and keeps no memory between inputs. An RNN has a loop: its output at one step is fed back in at the next step. That loop lets it handle inputs of any length and take earlier inputs into account.

Can an RNN handle sequences of any length?

In principle yes, since the same cell runs once per element. In practice a plain RNN loses information from more than a few dozen steps back because of vanishing gradients and the fixed-size state. LSTMs stretch this to hundreds of steps, and attention removes the bottleneck by looking back at every position directly.

Why do RNNs use tanh instead of ReLU?

The same weights are applied again at every step, so an unbounded activation like ReLU can let the hidden state grow out of control on long sequences. Tanh bounds each value between −1 and +1 and is centered at zero. ReLU RNNs can work with careful initialization, such as an identity W_h, but tanh is the default.

Are RNNs still used today?

Transformers have replaced them for most language tasks, but recurrent models are still used for streaming audio, on-device keyboards, and sensor data where memory and latency are tight. Recent state space models such as Mamba also keep a recurrent state, which lets them generate each new token at a fixed cost.

Related explainers

LSTMs — Gates and memory cells for long sequences
Vanishing & Exploding Gradients — Why deep networks struggle and how to fix it
Backpropagation — How networks learn from their mistakes
Seq2Seq & Beam Search — Encode an input sequence and decode an output — the architecture behind translation
Embeddings — Turning words and categories into vectors
The Transformer Architecture — A visual guide to 'Attention Is All You Need'

References

Last updated October 25, 2026