GloVe: Global Vectors for Word Representation

While Word2Vec models like Skip-gram and CBOW learn embeddings by predicting local context words within a sliding window, they don't directly leverage the extensive amount of statistical information present across the entire corpus. Methods like Latent Semantic Analysis (LSA) do use global statistics (matrix factorization on term-document or term-term matrices), but they often perform relatively poorly on tasks like word analogy, which measure finer semantic relationships.

GloVe, standing for Global Vectors for Word Representation, developed by Jeffrey Pennington, Richard Socher, and Christopher D. Manning at Stanford, aims to bridge this gap. It's designed to capture the best of both worlds: the meaning-capturing capabilities demonstrated by local context prediction methods (like Word2Vec) and the statistical power derived from global matrix factorization techniques.

The Core Idea: Ratios of Co-occurrence Probabilities

The central intuition behind GloVe is that ratios of word-word co-occurrence probabilities hold meaningful information. Consider the co-occurrence probabilities for words related to "ice" and "steam".

Let $P(k | w) = P_{wk}$ be the probability that word $k$ appears in the context of word $w$. Now, let's look at the ratio of these probabilities for different probe words $k$.

• Take $w_i = \text{ice}$ and $w_j = \text{steam}$.
• Consider a probe word $k = \text{solid}$. We expect the ratio $P(\text{solid} | \text{ice}) / P(\text{solid} | \text{steam})$ to be large, because "solid" is much more related to "ice".
• Consider $k = \text{gas}$. We expect the ratio $P(\text{gas} | \text{ice}) / P(\text{gas} | \text{steam})$ to be small, as "gas" relates more to "steam".
• Consider $k = \text{water}$ (related to both) or $k = \text{fashion}$ (related to neither). We expect the ratio $P(k | \text{ice}) / P(k | \text{steam})$ to be close to 1.

GloVe hypothesizes that these ratios encode the relationships between words, and the goal is to learn word vectors that capture these ratios.

The Co-occurrence Matrix

GloVe starts by constructing a large word-word co-occurrence matrix, denoted by $X$. An entry $X_{ij}$ in this matrix represents the number of times word $j$ (the context word) appears within a specific context window of word $i$ (the target word) across the entire corpus.

The definition of the "context window" is similar to that used in Word2Vec. Often, the contribution of a co-occurring word pair is weighted based on the distance between the words, with closer words contributing more. For instance, a weighting function like $1/d$ (where $d$ is the distance) might be used.

Let $X_i = \sum_k X_{ik}$ be the total number of times any word appears in the context of word $i$. Then, the probability of seeing word $j$ in the context of word $i$ is $P_{ij} = P(j | i) = X_{ij} / X_i$.

A representation of how word co-occurrences from a text corpus contribute to the entries in the co-occurrence matrix $X$.

The GloVe Model and Objective Function

GloVe aims to learn vectors for each word such that their dot product relates directly to their probability of co-occurrence. The model starts with the general idea that the relationship between three words $i$, $j$, and $k$ can be modeled using a function $F$ acting on their word vectors:

$F(w_i, w_j, \tilde{w}_k) = \frac{P_{ik}}{P_{jk}}$

Here, $w_i$ and $w_j$ are the vectors for the primary words, and $\tilde{w}_k$ is a separate context word vector for the probe word $k$. Using two sets of vectors ($w$ and $\tilde{w}$) makes the model more effective and easier to train, capturing the asymmetry in co-occurrence (e.g., $P(\text{solid} | \text{ice})$ is not necessarily the same as $P(\text{ice} | \text{solid})$).

Through mathematical derivation involving properties of vector differences and homomorphisms, the GloVe authors arrive at a specific form relating the dot product of word vectors to the logarithm of their co-occurrence count:

$w_i^T \tilde{w}_j + b_i + \tilde{b}_j \approx \log(X_{ij})$

Here, $b_i$ and $\tilde{b}_j$ are scalar bias terms for the target word $i$ and context word $j$, respectively. These biases help capture frequency effects independent of the vector interactions.

This relationship forms the basis of the GloVe objective function, which is a weighted least squares regression model:

$J = \sum_{i,j=1}^{V} f(X_{ij}) (w_i^T \tilde{w}_j + b_i + \tilde{b}_j - \log(X_{ij}))^2$

• $V$ is the vocabulary size.
• $w_i$ and $b_i$ are the vector and bias for target word $i$.
• $\tilde{w}_j$ and $\tilde{b}_j$ are the vector and bias for context word $j$.
• $X_{ij}$ is the co-occurrence count.
• $f(X_{ij})$ is a weighting function.

The weighting function $f(X_{ij})$ is important. It serves two main purposes:

1. It prevents $\log(X_{ij})$ from being undefined when $X_{ij} = 0$. In practice, $f(0) = 0$.
2. It down-weights very frequent co-occurrences (like "the" appearing with "a"), preventing them from dominating the training objective. It also down-weights extremely rare co-occurrences, which might be noisy or unreliable.

A commonly used weighting function is:

$f(x) = \begin{cases} (x / x_{\text{max}})^\alpha & \text{if } x < x_{\text{max}} \\ 1 & \text{otherwise} \end{cases}$

where $x_{\text{max}}$ is a threshold (e.g., 100) and $\alpha$ is typically set to $3/4$.

The model is trained using techniques like AdaGrad to minimize the objective function $J$, learning the optimal values for the word vectors $w_i$, context vectors $\tilde{w}_j$, and biases $b_i, \tilde{b}_j$. Often, the final representation for a word $i$ is taken as the sum $w_i + \tilde{w}_i$.

GloVe vs. Word2Vec

• Training Data: Word2Vec learns from local context windows streamed from the corpus. GloVe first computes global co-occurrence statistics across the entire corpus and then trains on these statistics.
• Objective: Word2Vec uses prediction-based objectives (predicting context words or the center word). GloVe uses a weighted least-squares objective to directly capture co-occurrence probabilities.
• Performance: Both methods produce high-quality word embeddings. GloVe often trains faster on large corpora because the expensive co-occurrence counting is done only once upfront. Performance on downstream tasks varies; sometimes GloVe excels, sometimes Word2Vec does. GloVe's explicit use of global statistics can be advantageous for capturing certain types of relationships.

Using GloVe Embeddings

Like Word2Vec, pre-trained GloVe vectors are widely available, trained on massive datasets like Wikipedia or Common Crawl. These pre-trained vectors often provide a strong starting point for various NLP tasks, saving significant computational effort compared to training from scratch. You can load these vectors and use them as input features for models handling tasks like text classification, sentiment analysis, or named entity recognition, similar to how you would use pre-trained Word2Vec embeddings.

In the next sections, we will look at how to visualize these high-dimensional embeddings and discuss practical ways to load and use pre-trained models.