ApX logoApX logo
ModelsCalculatorHardwareCoursesPricing
ApX logoApX logo
ModelsCalculatorHardwareCoursesPricing

All Courses

Introduction to Multimodal AI
Chapter 1: What is Multimodal AI?
Artificial Intelligence: A Brief Overview
Understanding Data Modalities: Text, Images, Audio
Defining Multimodal AI: Processing Diverse Data
Benefits of Combining Multiple Modalities
Multimodal vs. Unimodal AI: Core Differences
Examples of Multimodal Systems
Fundamental Challenges in Multimodal AI
An Illustrative Multimodal Task: Generating Image Descriptions
Practice: Identifying Modalities in Common Technologies
Quiz for Chapter 1
Chapter 2: Data Foundations for Multimodal Systems
Text Data Representation: From Characters to Meaning
Image Data Representation: Pixels, Features, and Structure
Audio Data Representation: Sound Waves to Digital Signals
Video Data: Sequences of Images and Sound
Basic Preprocessing for Different Data Types
Aligning Data from Multiple Sources
Comparing Information Across Modalities
Hands-on Practical: Observing Data Formats
Quiz for Chapter 2
Chapter 3: Techniques for Integrating Modalities
Approaches to Multimodal Fusion: Early, Intermediate, Late
Early Fusion: Combining Data at the Input Stage
Intermediate Fusion: Merging Processed Features
Late Fusion: Combining Independent Predictions
Shared Representations: Learning Common Features
Coordinated Representations: Mapping Between Modalities
Basic Architectures for Multimodal Learning
Introduction to Attention: Focusing on Relevant Information
Practice: Visualizing Fusion Methods
Quiz for Chapter 3
Chapter 4: Components of Multimodal AI Models
Extracting Features from Text Data
Extracting Features from Image Data
Extracting Features from Audio Data
Simple Neural Network Layers for Multimodal Tasks
Measuring Performance: Loss Functions for Combined Data
Training Multimodal Systems: An Overview
Basic Evaluation Metrics for Multimodal Outputs
Hands-on Practical: Conceptualizing a Simple Model
Quiz for Chapter 4
Chapter 5: Introductory Applications of Multimodal AI
Image Captioning Systems: Generating Text from Images
Visual Question Answering: Interacting with Images Through Questions
Text-to-Image Synthesis: Creating Visuals from Descriptions (Introduction)
Speech Recognition Enhanced by Visual Cues (Introduction)
Multimodal Sentiment Analysis: Understanding Opinions from Multiple Cues
Inputs and Outputs in Multimodal Applications
Practice: Brainstorming a Multimodal Solution
Quiz for Chapter 5

Text Data Representation: From Characters to Meaning

AI systems require data to be in a numerical format to perform their tasks. While an image might be represented as a grid of pixel values, text, fundamentally a sequence of characters, also needs a similar transformation. How raw text, composed of letters, punctuation, and spaces, is converted into numerical structures that AI models can understand and learn from is a primary focus. This process is the first step in enabling machines to derive meaning from written language.

From Text to Tokens: Breaking It Down

The very first step in processing text is usually tokenization. Think of tokenization as breaking down a long sentence or paragraph into smaller, manageable pieces called "tokens." Most commonly, these tokens are words. For example, the sentence "AI is fun!" can be tokenized into:

  • "AI"
  • "is"
  • "fun"
  • "!"

Punctuation marks like "!" are often treated as separate tokens because they can carry meaning (e.g., indicating excitement or the end of a sentence). Sometimes, text is tokenized into sub-word units or even individual characters, depending on the specific requirements of the AI task. For beginners, word-level tokenization is a great starting point.

A simple illustration of tokenizing a sentence into words and punctuation.

Building a Vocabulary: The Dictionary for AI

Once we have a collection of tokens, the next step is to build a vocabulary. A vocabulary is essentially a list of all unique tokens found in our text data. Each unique token in this vocabulary is then assigned a unique numerical identifier (an integer ID). This is like creating a dictionary where each word (token) has a specific page number (ID).

For instance, if our entire dataset only contained the sentences "AI is fun" and "AI is great", our tokens would be "AI", "is", "fun", "great". Our vocabulary could look like this:

  • "AI": 0
  • "is": 1
  • "fun": 2
  • "great": 3

So, the sentence "AI is fun" would be represented numerically as the sequence [0, 1, 2].

An example showing how tokens from a sentence are mapped to numerical IDs based on a defined vocabulary.

This numerical sequence is a basic way to represent text, but AI models often benefit from more structured vector representations.

Vectorizing Tokens: Giving Numbers More Structure

While sequences of IDs are a start, we often convert these tokens or sequences of tokens into vectors. A vector is simply an array of numbers. There are several common methods to do this.

One-Hot Encoding

One-Hot Encoding (OHE) is a straightforward technique. For each token, we create a vector that is as long as our entire vocabulary. This vector is filled with zeros, except for a single '1' at the index corresponding to the token's ID in the vocabulary.

Imagine our vocabulary is {"the": 0, "cat": 1, "sat": 2}.

  • The token "cat" (ID: 1) would be represented as the vector [0, 1, 0].
  • The token "the" (ID: 0) would be [1, 0, 0].

A token 'cat' represented as a one-hot encoded vector based on a small vocabulary. The '1' appears at the index assigned to 'cat'.

While simple, OHE has drawbacks:

  1. High Dimensionality: If your vocabulary has 20,000 words, each word's vector will have 20,000 dimensions, mostly zeros. This is computationally expensive.
  2. Sparsity: The vectors are "sparse" (mostly zeros).
  3. No Semantic Relationship: The OHE vectors for "cat" and "dog" are just as different from each other as "cat" and "car." The representation doesn't capture that "cat" and "dog" are semantically more similar (both are animals).

Bag-of-Words (BoW)

The Bag-of-Words (BoW) model is another popular way to represent an entire piece of text (like a sentence or document) as a single vector. This vector has one entry for each word in the vocabulary. Each entry can be:

  • A binary value (0 or 1) indicating the presence or absence of a word.
  • A count of how many times each word appears in the text.

For example, with a vocabulary {"the", "cat", "sat", "on", "mat"}:

  • The sentence "the cat sat" would be [1, 1, 1, 0, 0] (if using counts).
  • The sentence "the cat the cat" would be [2, 2, 0, 0, 0].

Bag-of-Words representation for two sentences using word counts. Each row is a vector for a sentence.

The main limitation of BoW is that it loses all information about word order. "The cat sat on the mat" and "The mat sat on the cat" would have very similar BoW representations if they used the same words, even though their meanings are different.

A common refinement of BoW is TF-IDF (Term Frequency-Inverse Document Frequency). TF-IDF doesn't just count words; it also weighs how important a word is to a document within a larger collection of documents. It gives higher weights to words that are frequent in a specific document but rare across all documents. For a beginner's course, understanding that TF-IDF is a more sophisticated way to score words in the BoW model is sufficient.

Towards Meaning: An Introduction to Word Embeddings

The methods discussed so far (OHE, BoW) are useful, but they don't inherently capture the meaning or semantic relationships between words. This is where word embeddings come in.

Word embeddings represent words as dense, low-dimensional vectors (unlike the sparse, high-dimensional vectors of OHE). The magic of word embeddings is that words with similar meanings tend to have similar vector representations. These vectors are typically learned from large amounts of text data, and the process aims to place words in a "vector space" such that their geometric relationships reflect their semantic relationships.

For instance, in an ideal embedding space:

  • The vector for "king" might be close to the vector for "queen."
  • The vector for "apple" might be close to "orange" but far from "car."
  • A famous (though simplified) example is that the vector operation vector("king") - vector("man") + vector("woman") might result in a vector very close to vector("queen").

These vectors are "dense" because most of their values are non-zero, and "low-dimensional" typically means they have a few hundred dimensions, rather than tens of thousands like OHE.

A simplified 2D view where related words (like fruits or royalty) are closer to each other in the vector space. "Car" and "truck" form another cluster.

Word embeddings like Word2Vec, GloVe, and FastText are powerful because they provide a way for AI models to understand meaning and context, which is a significant step from just characters to actual meaning. For now, it's important to know that they exist and represent a more advanced way to turn words into numbers that carry semantic information.

By transforming text into these various numerical formats, we lay the groundwork for AI models to process and "understand" language. Each representation has its strengths and weaknesses, and the choice often depends on the specific task at hand. These numerical representations are the fundamental building blocks for any AI system that works with text data.

  • Speech and Language Processing (3rd ed. draft), Daniel Jurafsky and James H. Martin, 2025 - An authoritative and comprehensive textbook for natural language processing, covering text preprocessing, tokenization, and various word representation methods.
  • Efficient Estimation of Word Representations in Vector Space, Tomas Mikolov, Kai Chen, Greg Corrado, Jeffrey Dean, 2013 arXiv preprint DOI: 10.48550/arXiv.1301.3781 - The seminal paper introducing Word2Vec, a highly influential model for learning dense word embeddings that capture semantic similarities.
  • GloVe: Global Vectors for Word Representation, Jeffrey Pennington, Richard Socher, Christopher Manning, 2014 Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) (Association for Computational Linguistics) DOI: 10.3115/v1/D14-1162 - Presents GloVe, an unsupervised model for learning word embeddings based on global co-occurrence statistics, offering an alternative to Word2Vec.
  • Introduction to Information Retrieval, Christopher D. Manning, Prabhakar Raghavan and Hinrich Schütze, 2008 (Cambridge University Press) - A standard textbook for information retrieval, detailing fundamental text representation models like Bag-of-Words and TF-IDF.

© 2026 ApX Machine Learning