Understanding Data Modalities: Text, Images, Audio

Artificial Intelligence (AI) systems are fundamentally designed to process information, much like humans. This processing involves various kinds of input. Information itself appears in many forms, and these different forms or types of data are known as modalities in the AI domain.

Think of modalities as the different channels through which information can be received or expressed. For humans, these are our senses: sight, hearing, touch, taste, and smell. AI systems also work with information from various sources, and understanding these sources is fundamental before we can explore how AI combines them. Let's look at the primary data modalities you'll encounter in Multimodal AI.

Text: The Language We Write and Read

Text is perhaps the most familiar data modality. It's information conveyed through written language. This includes:

• The words on this webpage.
• Articles you read online or in newspapers.
• Messages you send via email or social media.
• Books, essays, and even computer code.

For an AI system, text isn't just a collection of letters. It's a sequence of characters, words, and sentences that carry meaning. While we'll get into the specifics of how AI processes text later, for now, think of it as structured information that AI can analyze to understand topics, sentiment, or even answer questions based on the provided text.

Images: Capturing Visual Perspectives

Image data represents visual information. It's how we capture and share what we see. Examples include:

• Photographs taken with your phone or camera.
• Drawings, paintings, and illustrations.
• Diagrams and charts.
• Medical scans like X-rays or MRIs.
• Satellite views of Earth.

When an AI system "looks" at an image, it typically sees a grid of tiny dots called pixels. Each pixel has color and brightness information. By analyzing patterns in these pixels, AI can learn to identify objects (like a cat or a car), recognize scenes (a beach, a forest), or detect specific features within an image.

Audio: The Sounds Around Us

Audio data is information conveyed through sound. This modality encompasses a wide range of sound types:

• Spoken language: Someone talking, a podcast, or voice commands given to a smart assistant.
• Music: Melodies, rhythms, and harmonies from various instruments or voices.
• Environmental sounds: A dog barking, a car horn, rain falling, or birds singing.

For an AI system, audio is typically represented as sound waves that have been converted into a digital format. These digital signals capture characteristics like frequency (pitch) and amplitude (loudness) over time. AI can analyze these signals to transcribe speech into text, identify different musical genres, or recognize specific sound events.

An AI system can receive input from various data modalities like text, images, and audio.

A Glimpse at Other Data Types

While text, images, and audio are the most common modalities we'll discuss in this introductory course, it's good to know that AI can work with other types of data too. For instance:

• Video: Often, video is considered a modality on its own, but it's essentially a sequence of image frames presented over time, frequently accompanied by an audio track. We'll touch upon this in more detail in the next chapter.
• Sensor Data: This can include readings from various sensors like temperature, pressure, GPS coordinates, or motion data from accelerometers (like those in your smartphone).
• Tabular Data: Data organized in rows and columns, like in a spreadsheet, is another form of information AI can process, though it's often handled by different specialized AI techniques.

For our purposes in "Introduction to Multimodal AI," we will primarily focus on how AI systems work with combinations of text, image, and audio data.

Why Understanding Modalities is Important

Each data modality carries information in a unique way and has its own inherent structure. Text is sequential, images are spatial, and audio unfolds over time. Recognizing these differences is the first step in appreciating the complexities and capabilities of Multimodal AI. When an AI system can process and relate information from these diverse sources, it can achieve a more comprehensive understanding, much like humans do.

In the sections that follow, we'll explore what it means to combine these modalities and why doing so can lead to more powerful and intelligent AI systems.