RNNs Remember by Carrying State

An RNN carries one compressed summary forward, and that compression is both the mechanism and the ceiling. The hidden state is the only thing that reaches the last step. Whatever the model knows about the past must fit inside it.

Images are spatial. Captions are sequential. A vision system that describes or acts over time needs memory. Recurrence is the old answer: read one token at a time and carry a state forward.

The hidden state is a notebook rewritten at every step

At each step the model reads the current input and the previous hidden state, then writes a new one. That is the whole recurrence. The notebook is rewritten at every step and never extended.

A character-level model shows the shape at its smallest. The input is one character. The output is a distribution over the next character. Every earlier character reaches that distribution only through the state.

That summary is not a transcript. A long-range dependency that fails is information the representation did not keep.

The same transition weights recur at every step

One transition function runs at every time step. The same weights read the first element of the sequence and the last.

Convolution reuses one filter across space. Recurrence reuses one transition across time. Both are bets about structure, and this bet is that the past compresses into a single state.

Sharing pays twice. It handles sequences of variable length, and it learns temporal patterns that repeat at any position. It also builds one long chain. A gradient that reaches the first step travels back through every step in between.

Backpropagation through time makes the chain explicit

To train the transition, unroll the network across time and run backprop on the unrolled graph. The same parameters appear at every step, so their gradients accumulate across the whole sequence.

That backward path is the chain rule the backprop note made concrete, now multiplied along time. Gradients vanish or explode as they pass through many repeated transitions.

The failure has one signature. The model keeps recent tokens and forgets older ones that still matter.

Gates buy a stable path for information across time

An LSTM splits the state in two. A memory cell holds what the model keeps. The rest of the network sees only the exposed hidden state.

Gates control the write, forget, and read operations, and the model learns each gate. It decides what enters the cell and what decays inside it.

That is controlled memory, and it buys a more stable path for information and gradients across time. The model keeps important information longer and lets irrelevant noise decay.

Transformers displaced RNNs for many tasks, and the state lesson outlasted them.

Image captioning is the bridge from one label to a sequence

Image captioning starts with visual features from a CNN. A sequence model then produces words one step at a time.

The two parts have clean jobs. The encoder extracts visual evidence. The decoder turns that evidence and its own previous words into the next word.

The image features initialize or condition the decoder. The visual evidence enters once and must last to the final word.

That split is the frozen backbone with a new head from the transfer note, moved from space to time. Both reuse one visual representation for a new output.

The Builder Test

Write down what the hidden state must still hold at the last step. Name a specific item.

Then count the steps between where that information enters and where the model uses it. That count is the length of the trip the state must survive.

Build the smallest version of that trip. Make a sequence whose answer depends on one early step, then read the output at the last step.

A model that holds on short sequences and fails on long ones has reported the answer. The state dropped the early item.

Check the objective too. A loss scored one step at a time can stay low while the whole caption is wrong.

A weak review says the model has memory. The strong review names which memory is too compressed to survive.

What Carries

Whenever a model reads a sequence, name what it carries forward and what it drops before the decision. The dropped part is where the failure starts.

Gates lengthen the trip a state can survive. The bottleneck is still one state, and the next note removes it instead of enlarging it.