ApX logoApX logo
ModelsCalculatorHardwareCoursesPricing
ApX logoApX logo
ModelsCalculatorHardwareCoursesPricing

All Courses

Natural Language Processing Fundamentals
Chapter 1: NLP Concepts and Advanced Text Processing
The Natural Language Processing Pipeline
Advanced Tokenization Methods
Stemming and Lemmatization Compared
Handling Noise in Text Data
Advanced Stop Word Customization
Text Normalization Techniques
Hands-on Practical: Building Preprocessing Pipelines
Quiz for Chapter 1
Chapter 2: Feature Engineering for Text Representation
From Bag-of-Words to TF-IDF
Calculating TF-IDF Scores
Using N-grams to Capture Context
Introduction to Feature Hashing
Dimensionality Reduction for Text Features
Comparing Different Text Representation Methods
Hands-on Practical: Generating Text Features
Quiz for Chapter 2
Chapter 3: Supervised Learning for Text Classification
Classification Algorithms Review
Applying Classifiers to Text Data
Model Evaluation Metrics for Classification
Cross-Validation Strategies
Hyperparameter Tuning for Text Models
Addressing Imbalanced Datasets
Practice: Building a Text Classifier
Quiz for Chapter 3
Chapter 4: Understanding Sequential Data with Embeddings
Limitations of Frequency-Based Models
Introduction to Distributional Semantics
Word Embedding Concepts
Word2Vec: CBOW and Skip-gram Architectures
GloVe: Global Vectors for Word Representation
Visualizing Word Embeddings
Using Pre-trained Word Embedding Models
Hands-on Practical: Working with Word Embeddings
Quiz for Chapter 4
Chapter 5: Introduction to Sequence Models for NLP
The Need for Sequence Awareness
Recurrent Neural Network (RNN) Basics
Understanding the Vanishing Gradient Problem
Long Short-Term Memory (LSTM) Networks
Gated Recurrent Units (GRUs)
Applying Sequence Models to Text
Hands-on Practical: Building a Simple Sequence Model
Quiz for Chapter 5

Long Short-Term Memory (LSTM) Networks

Recurrent Neural Networks (RNNs) face challenges in learning long-range dependencies within sequences. A primary issue is the vanishing gradient problem, which causes gradients during backpropagation to become too small to effectively update the network's weights for earlier time steps. This limitation hinders their ability to capture relationships between distant elements. To address this, a more advanced recurrent architecture was developed: the Long Short-Term Memory (LSTM) network, first introduced by Hochreiter and Schmidhuber in 1997.

LSTMs are explicitly designed to remember information for long periods. The central innovation is the cell state (CtC_tCt​), often seen as the network's "memory," which can carry information relatively unchanged across many time steps. Unlike the single hidden state transformation in a simple RNN, LSTMs use a system of gates to carefully regulate the flow of information into and out of this cell state.

The LSTM Cell Structure

An LSTM unit processes data sequentially, taking the input at the current time step (xtx_txt​) and the hidden state from the previous time step (ht−1h_{t-1}ht−1​) to compute the new hidden state (hth_tht​) and update its internal cell state (CtC_tCt​). This process is controlled by three primary gates:

  1. Forget Gate (ftf_tft​): This gate decides what information to discard from the cell state. It looks at ht−1h_{t-1}ht−1​ and xtx_txt​ and outputs a number between 0 and 1 for each number in the previous cell state Ct−1C_{t-1}Ct−1​. A 1 represents "completely keep this," while a 0 represents "completely get rid of this." It uses a sigmoid activation function (σ\sigmaσ). ft=σ(Wf[ht−1,xt]+bf)f_t = \sigma(W_f [h_{t-1}, x_t] + b_f)ft​=σ(Wf​[ht−1​,xt​]+bf​) Here, [ht−1,xt][h_{t-1}, x_t][ht−1​,xt​] denotes the concatenation of the previous hidden state and the current input, WfW_fWf​ are the weights, and bfb_fbf​ is the bias for the forget gate.

  2. Input Gate (iti_tit​): This gate determines which new information will be stored in the cell state. It consists of two parts:

    • First, a sigmoid layer decides which values to update (iti_tit​). it=σ(Wi[ht−1,xt]+bi)i_t = \sigma(W_i [h_{t-1}, x_t] + b_i)it​=σ(Wi​[ht−1​,xt​]+bi​)
    • Second, a tanh layer creates a vector of new candidate values, C~t\tilde{C}_tC~t​, that could be added to the state. C~t=tanh⁡(WC[ht−1,xt]+bC)\tilde{C}_t = \tanh(W_C [h_{t-1}, x_t] + b_C)C~t​=tanh(WC​[ht−1​,xt​]+bC​) These two parts are combined to update the cell state.
  3. Updating the Cell State: The old cell state Ct−1C_{t-1}Ct−1​ is updated to the new cell state CtC_tCt​. We multiply the old state by ftf_tft​, forgetting the things we decided to forget earlier. Then we add it∗C~ti_t * \tilde{C}_tit​∗C~t​. This is the new candidate information, scaled by how much we decided to update each state value. Ct=ft∗Ct−1+it∗C~tC_t = f_t * C_{t-1} + i_t * \tilde{C}_tCt​=ft​∗Ct−1​+it​∗C~t​ Note the use of addition here. This additive interaction is important for allowing gradients to flow better compared to the repeated multiplications in simple RNNs.

  4. Output Gate (oto_tot​): This gate decides what the next hidden state hth_tht​ (the output for this time step) should be. The output will be based on the filtered cell state.

    • First, a sigmoid layer determines which parts of the cell state to output. ot=σ(Wo[ht−1,xt]+bo)o_t = \sigma(W_o [h_{t-1}, x_t] + b_o)ot​=σ(Wo​[ht−1​,xt​]+bo​)
    • Then, the (now updated) cell state CtC_tCt​ is passed through tanh (to push values between -1 and 1) and multiplied by the output of the sigmoid gate oto_tot​. This ensures we only output the parts we decided to. ht=ot∗tanh⁡(Ct)h_t = o_t * \tanh(C_t)ht​=ot​∗tanh(Ct​)

The hidden state hth_tht​ is then passed to the next time step (as ht−1h_{t-1}ht−1​) and can also be used as the output of the model at the current step.

Visualizing the LSTM Cell

The structure of an LSTM cell, with its gates controlling the flow of information through the cell state, can be visualized as follows:

Data flow within a single LSTM cell. Arrows indicate data dependencies. Circles with 'X' denote element-wise multiplication, '+' denotes element-wise addition, 'σ' denotes the sigmoid function, and 'tanh' denotes the hyperbolic tangent function. The cell state acts as a conveyor belt, with gates controlling information removal and addition.

How LSTMs Mitigate Vanishing Gradients

The mechanism allowing LSTMs to handle long-range dependencies better than simple RNNs lies in the cell state and the gating mechanism.

  • Additive Updates: The cell state CtC_tCt​ is updated primarily through addition (Ct=ft∗Ct−1+it∗C~tC_t = f_t * C_{t-1} + i_t * \tilde{C}_tCt​=ft​∗Ct−1​+it​∗C~t​). This additive interaction makes it easier for gradients to flow backward through time without vanishing as quickly as they do through the repeated matrix multiplications found in simple RNNs.
  • Gate Control: The gates act as regulators. The forget gate can learn to preserve important information from the distant past by keeping its activation close to 1 for certain elements of the cell state. Similarly, the input gate can learn to prevent irrelevant new information from corrupting the cell state. This selective memory allows the network to maintain context over long sequences.

Think of the cell state as a conveyor belt carrying information through time. The forget gate removes items from the belt, the input gate adds new items, and the output gate reads items off the belt to decide the immediate output (hidden state). This structure is much more effective at preserving signals over long durations.

LSTMs have been highly successful in various NLP tasks, including machine translation, sentiment analysis, and sequence labeling, precisely because of their ability to capture long-range context. While they are more complex than simple RNNs, their effectiveness often justifies the additional computational overhead.

In the next section, we'll look at Gated Recurrent Units (GRUs), a slightly simpler variant of the LSTM that often achieves comparable performance.

  • Long Short-Term Memory, Sepp Hochreiter, Jürgen Schmidhuber, 1997 Neural Computation, Vol. 9 (MIT Press) DOI: 10.1162/neco.1997.9.8.1735 - The original academic paper introducing the Long Short-Term Memory (LSTM) network.
  • Deep Learning, Ian Goodfellow, Yoshua Bengio, Aaron Courville, 2016 (MIT Press) - A comprehensive textbook providing a detailed theoretical explanation of LSTMs within the context of recurrent neural networks.

© 2026 ApX Machine Learning