Practice: Identifying Modalities in Common Technologies

This exercise focuses on identifying data modalities in common technologies. Understanding these different types of data, which Multimodal AI aims to process together, is a primary step. This practice will help you become more observant of the various types of data that everyday technologies handle, a fundamental skill before considering how an AI system might process them.

Your Task: Become a Modality Detective

Your goal is to look at some common technologies you likely use or are familiar with. For each one, think about:

1. What kinds of information do you provide to it (inputs)?
2. What kinds of information does it provide back to you (outputs)?
3. Based on these inputs and outputs, what are the primary data modalities involved?

Let's work through a few examples together. Try to think them through yourself before reading our analysis.

Example 1: A Smart Speaker (like Amazon Echo or Google Home)

• Think about it: How do you interact with it? What does it do in response?

• Our Analysis:

  • Inputs: Primarily, your voice commands (audio modality). You might also press physical buttons, which is a form of interaction, though we typically focus on the rich data types AI processes.
  • Outputs: The speaker responds with synthesized speech (audio modality). It might also show light patterns on the device (visual modality) to indicate it's listening or processing.
  • Modalities Involved: The main ones are Audio (input and output) and Visual (simple output).

This is a classic example of a system that, at its core, processes audio information but often uses simple visual cues.

Example 2: Video Conferencing Software (like Zoom or Microsoft Teams)

• Think about it: What are all the ways you and others share information during a video call?

• Our Analysis:

  • Inputs: Your live video feed from your camera (video modality), your voice picked up by your microphone (audio modality), text messages you type in the chat (text modality), and possibly screen sharing (which is essentially a sequence of images, so video or image sequence modality).
  • Outputs: The video feeds of other participants (video modality), their voices (audio modality), text messages in chat (text modality), and shared screens (video/image sequence modality).
  • Modalities Involved: Video, Audio, and Text are the prominent modalities here.

Video conferencing is inherently multimodal, combining sight, sound, and written communication.

Example 3: A Social Media App Focused on Visuals (like Instagram or Pinterest)

• Think about it: When you use this app, what do you upload? What do you see and interact with?

• Our Analysis:

  • Inputs: You might upload photos (image modality) or videos (video modality, which also contains an audio modality component). You write captions, comments, or messages (text modality). You might also record or react with audio snippets in some features.
  • Outputs: You see images and videos shared by others. You read text in captions, comments, and user profiles. You hear audio from videos.
  • Modalities Involved: Image, Video, Audio, and Text.

These platforms are rich in different types of media, making them prime examples of multimodal information environments.

Example 4: A Food Delivery App

• Think about it: How do you find a restaurant? How do you place an order? What information does the app provide?

• Our Analysis:

  • Inputs: You type search queries or browse categories (text modality). You tap on images of food or restaurants (interacting with the visual modality). You might provide your location (location data, which can be seen as another type of modality).
  • Outputs: The app displays restaurant listings (text modality), food menus (text modality), photos of dishes (image modality), and maps showing restaurant locations or delivery progress (visual/graphical modality, often incorporating location data).
  • Modalities Involved: Text, Image, Location Data, and Graphical Visuals (maps).

Even an app that seems straightforward like food delivery relies on multiple types of data to function effectively.

Now It's Your Turn

Think about two or three other pieces of technology you use regularly. It could be:

• Your smartphone's operating system.
• A modern car's infotainment system.
• A gaming console.
• An e-learning platform.
• A news website or app.

For each one, jot down:

1. The Technology:
2. Primary Inputs (and their modalities):
3. Primary Outputs (and their modalities):
4. List of Identified Modalities:

Reflect on Your Findings

Once you've analyzed a few technologies, consider these questions:

• Were you surprised by how many modalities are involved in seemingly simple applications?
• For any given technology, how do the different modalities work together to provide a complete experience? For instance, in a navigation app, how do maps (visual), voice instructions (audio), and place names (text) combine?
• Could any of these technologies function effectively with only a single modality? What would be lost?

This exercise isn't just about listing data types. It's about starting to see through the lens of multimodal information. As we progress through this course, you'll learn how AI systems are designed to understand and generate these different forms of data, often in a coordinated way, similar to how humans perceive and interact. Recognizing these modalities in existing technology is the first step toward appreciating the complexity and potential of Multimodal AI.