Extracting Features from Text Data

To effectively build multimodal AI models, transforming raw data from each modality into a format that algorithms can understand and process is essential. For text, this involves converting strings of characters, words, and sentences into numerical representations, often called features or embeddings. Raw text itself isn't directly usable by most machine learning models, which expect numerical input. Common techniques for extracting these meaningful features from text data are discussed.

The Need for Numerical Representation

Imagine trying to teach a computer to understand a sentence like "A fluffy cat sleeps on a warm rug." The computer doesn't inherently know what "fluffy," "cat," or "sleeps" means. It only sees a sequence of characters. Feature extraction is the process of converting this text into a set_of_numbers (a vector) that captures some aspect of its meaning or structure. These numerical vectors can then be fed into machine learning models, which learn patterns from them. In a multimodal system, these text features will later be combined with features extracted from images, audio, or other data types.

A Simple Starting Point: Bag-of-Words (BoW)

One of the most straightforward ways to represent text numerically is the Bag-of-Words (BoW) model. Think of it like this: you take all the words in your document, throw them into a "bag," and count how many times each word appears. The order of words and the grammar are disregarded; only the presence and frequency matter.

Here's how it generally works:

1. Create a Vocabulary: First, you gather all your text documents (your corpus) and compile a list of all unique words that appear in them. This list is your vocabulary.
2. Vectorize Documents: For each document, you create a numerical vector. The length of this vector is equal to the size of your vocabulary. Each position in the vector corresponds to a unique word in the vocabulary.
3. Count Frequencies: The value at each position in the vector is typically the count of how many times the corresponding word from the vocabulary appears in that specific document.

Let's consider a tiny example:

• Document 1: "The cat sat on the mat."
• Document 2: "The dog chased the cat."

Our vocabulary (ignoring case and punctuation for simplicity, and using unique words) might be: {"the", "cat", "sat", "on", "mat", "dog", "chased"}.

• BoW vector for Document 1:

  • "the": 2, "cat": 1, "sat": 1, "on": 1, "mat": 1, "dog": 0, "chased": 0
  • So, the vector could be [2, 1, 1, 1, 1, 0, 0]

• BoW vector for Document 2:

  • "the": 2, "cat": 1, "sat": 0, "on": 0, "mat": 0, "dog": 1, "chased": 1
  • So, the vector could be [2, 1, 0, 0, 0, 1, 1]

Strengths of BoW:

• Simplicity: It's easy to understand and implement.
• Effectiveness for some tasks: It can work surprisingly well for tasks like document classification where the presence of certain keywords is a strong indicator.

Weaknesses of BoW:

• Loss of Word Order: "Man bites dog" and "Dog bites man" would have very similar BoW representations (if not identical, depending on exact vocabulary construction), even though their meanings are opposite.
• Loss of Context: The meaning of words can change based on surrounding words (e.g., "bank" of a river vs. "bank" for money). BoW doesn't capture this.
• Sparsity: For large vocabularies, most document vectors will be filled with zeros, making them sparse. This can be computationally inefficient.
• Vocabulary Size: The vocabulary can become extremely large, leading to very high-dimensional vectors.

Adding Importance: Term Frequency-Inverse Document Frequency (TF-IDF)

The Bag-of-Words model treats all words equally based on their raw counts. However, some words are inherently more informative than others. For instance, words like "the," "a," or "is" appear very frequently in almost all English texts but carry little specific meaning about the document's content. TF-IDF tries to address this by giving higher weights to words that are frequent in a particular document but rare across the entire collection of documents (the corpus).

TF-IDF is a product of two statistics: Term Frequency and Inverse Document Frequency.

1. Term Frequency (TF): This measures how often a term (word) $t$ appears in a document $d$. There are several ways to calculate TF. A simple one is the raw count, but often it's normalized to prevent a bias towards longer documents:

   $$TF(t, d) = \frac{\text{Number of times term } t \text{ appears in document } d}{\text{Total number of terms in document } d}$$

2. Inverse Document Frequency (IDF): This measures how important a term is across the entire corpus $D$. It scales down the weight of terms that appear in many documents and scales up the weight of terms that appear in few documents. A common way to calculate IDF is:

   $$IDF(t, D) = \log \left( \frac{\text{Total number of documents } |D|}{\text{Number of documents containing term } t} \right)$$

   To avoid division by zero if a term isn't in any document (which shouldn't happen if the vocabulary is built from the corpus) or a term is in all documents (leading to $\log(1)=0$), smoothing is often applied, for instance, by adding 1 to the denominator: $IDF(t, D) = \log \left( \frac{|D|}{1 + \text{Number of documents containing term } t} \right) + 1$. For our introductory purposes, the basic idea is important.

The TF-IDF score for a term $t$ in document $d$ within corpus $D$ is then:

$$TFIDF(t, d, D) = TF(t, d) \times IDF(t, D)$$

A high TF-IDF score is achieved by a high term frequency (the term is common in the specific document) and a low document frequency of the term in the whole collection of documents (the term is rare overall). This makes TF-IDF effective at highlighting words that are characteristic of a particular document.

For example, in a collection of news articles, the word "election" might have a high TF-IDF score in an article specifically about an election, while the word "today" would likely have a low IDF score (and thus a lower TF-IDF score) because it appears in many articles.

Strengths of TF-IDF:

• Reduces impact of common words: Better than BoW at identifying significant words.
• Simple to compute: Still relatively straightforward.

Weaknesses of TF-IDF:

• Still loses word order and context: Like BoW, it doesn't understand grammar or semantic relationships between words.
• Sparsity: Vectors can still be sparse, though the values are now weighted.

Capturing Meaning: An Introduction to Word Embeddings

While BoW and TF-IDF provide useful numerical representations, they don't capture the meaning or semantic relationships between words. For example, "happy" and "joyful" are synonyms, but in a BoW or TF-IDF model, their vectors might be completely different and unrelated if they are treated as distinct vocabulary items.

Word embeddings aim to solve this. They represent words as dense, low-dimensional vectors (e.g., 50 to 300 dimensions, compared to potentially tens of thousands for BoW/TF-IDF). The main aspect of these embeddings is that words with similar meanings are represented by vectors that are close to each other in this vector space.

Imagine a space where words like "king," "queen," "prince," and "princess" are located. Word embeddings learn representations suchthat the vector relationship between "king" and "queen" might be similar to the vector relationship between "man" and "woman." This is often illustrated by the famous example: vector("king") - vector("man") + vector("woman") ≈ vector("queen").

How are they different from BoW/TF-IDF?

• Dense vs. Sparse: Embeddings are dense vectors (most values are non-zero), while BoW/TF-IDF vectors are sparse (mostly zeros).
• Low vs. High Dimensionality: Embeddings typically have a few hundred dimensions, while BoW/TF-IDF can have dimensions equal to the vocabulary size, which can be very large.
• Meaning vs. Counts: Embeddings are trained to capture semantic relationships, while BoW/TF-IDF are based on word co-occurrence counts at the document level.

Popular algorithms for creating word embeddings include:

• Word2Vec (developed at Google)
• GloVe (Global Vectors for Word Representation) (developed at Stanford)
• FastText (developed at Facebook)

A significant advantage is the availability of pre-trained word embeddings. Researchers have trained these models on large amounts of text data (like all of Wikipedia or large news corpora). This means you can often download these pre-trained embeddings and use them directly in your models without needing to train them from scratch on your own (potentially smaller) dataset. This is useful, especially when your dataset isn't large enough to learn high-quality embeddings on its own.

For multimodal systems, these dense word (or sentence) embedding vectors serve as rich, semantically informed inputs to the parts of your neural network that will later combine this textual information with features from images or audio.

From Words to Sentences and Documents

So far, we've mostly talked about representing individual words. But often, we need to represent entire sentences or documents. How do we do that?

• With BoW or TF-IDF, the representation is already at the document level.
• With word embeddings, you have vectors for individual words. To get a single vector for a sentence or document, common approaches include:
  • Averaging: Simply take the average of the embedding vectors of all words in the sentence/document. This is a simple but often effective baseline.
  • Weighted Averaging: Average word embeddings, but give more weight to important words (e.g., using their TF-IDF scores as weights).
  • More Advanced Methods: Use neural network architectures like Recurrent Neural Networks (RNNs, particularly LSTMs or GRUs) or Transformers. These models can process sequences of word embeddings and learn to produce a single vector representation for the entire sequence, capturing word order and context more effectively. For a beginner's course, just knowing these exist is a good starting point.

The Goal: Ready for Integration

Regardless of the technique chosen, BoW, TF-IDF, or word embeddings, the objective of text feature extraction is the same: to convert raw text into a set of numerical features. These features are the building blocks that our AI model can understand. Once we have these numerical representations for text, and similarly for other modalities like images and audio (which we'll cover next), we're one step closer to combining them in a multimodal AI system. These extracted features are the inputs that will be fed into the integration techniques and model architectures we discussed in the previous chapter.