Practice: Brainstorming a Multimodal Solution

Multimodal AI offers impressive capabilities, such as generating image descriptions, answering questions about pictures, and creating images from text. These applications combine different types of information, or modalities, to perform tasks that would be difficult or impossible with just one type of data.

Now it's your turn to put on your designer hat! This practice exercise is about brainstorming your own simple multimodal AI application. Don't worry about the technical details of how to build it; the goal is to think creatively about how combining modalities could solve a problem or create something new and interesting.

Your Turn: Design a Multimodal Helper

Let's walk through a thinking process to help you sketch out an idea. Grab a piece of paper or open a new document, and let's get started.

1. Identify a Scenario or Need

Think about your daily life, hobbies, or simple tasks.

• Is there anything that feels a bit clunky or could be improved with a little "smart" assistance?
• Could an AI that understands more than one type of information at once be helpful?

For instance, imagine you're trying to assemble a new piece of furniture, and the instructions are a bit confusing. Or perhaps you're learning a new language and want to practice pronunciation while looking at related images.

Write down one or two scenarios that come to mind.

2. What Kinds of Information are Involved? (The Modalities)

For the scenario you've chosen, what types of data are naturally present or would be useful for an AI to access?

• Visuals? (Images, videos, what someone is looking at)
• Sounds? (Speech, music, environmental noises)
• Text? (Written instructions, questions, labels)

Consider the examples from this chapter:

• Image captioning uses an image (input) to produce text (output).
• Visual Question Answering uses an image and text (a question) as input to produce text (an answer) as output.
• Multimodal sentiment analysis might use video (visual expressions), audio (tone of voice), and text (comments) to understand an opinion.

For your chosen scenario, list the main modalities your AI helper would need to understand or generate.

3. What Goes In? What Comes Out? (Inputs and Outputs)

Now, let's get more specific.

• Inputs: What exact pieces of information would a user (or the environment) provide to your AI system?
  • Example: If your idea is a "Plant Identifier," the input might be an image of a plant taken with a phone and a spoken question like, "What is this plant and how do I care for it?"
• Outputs: What would the AI system produce as a result?
  • Example (continuing Plant Identifier): The output could be the plant's name displayed as text, accompanied by spoken care instructions.

Describe the inputs and outputs for your multimodal AI idea.

4. How Do the Modalities Work Together?

This is where the "multimodal" aspect really shines. How does the combination of different data types help your AI achieve its goal?

• Does one modality provide context for another?
• Are they processed together to get a richer understanding?
• Does the AI translate information from one modality to another?

Think about the "Speech Recognition Enhanced by Visual Cues" example. The AI doesn't just listen to audio; it also watches lip movements (visual). The audio and visual information work together to improve the accuracy of speech recognition, especially in noisy environments.

Briefly explain how the different modalities in your idea would interact or complement each other.

5. Why is a Multimodal Approach Better?

Consider if your task could be done with just one modality. If so, what are the advantages of using multiple modalities?

• Does it make the system more accurate?
• More intuitive to use?
• More to different situations?
• Does it enable a completely new capability?

For image captioning, just having an image (unimodal) doesn't tell you its description. You need the AI to bridge the gap between visual information and textual language.

Jot down a sentence or two about why a multimodal approach is beneficial for your specific idea.

Let's try a quick example together:

• Scenario/Need: Help a tourist navigate a foreign city and understand signs or menus not in their native language.
• Modalities:
  • Input: Image (photo of a sign or menu via a phone camera).
  • Input: Text (the tourist's preferred language, set in an app).
  • Output: Text (translated text, overlaid on the image or displayed below).
  • Output: Audio (spoken translation of the text).
• Inputs & Outputs:
  • Input: User points phone camera at a sign (image), has pre-selected "English" as their language (text).
  • Output: The foreign text on the sign is translated into English and displayed on the phone screen (text), and optionally read aloud (audio).
• How Modalities Work Together: The system uses image processing to detect text in the visual input. This detected text is then processed by a translation model, which uses the target language (another text input) to produce the translated text output. The audio output is generated from this translated text.
• Why Multimodal is Better: A purely text-based translator would require the user to type the foreign text, which can be difficult with unfamiliar alphabets. A purely image-based system might not know what language to translate to. Combining image capture (for the source text) with a text setting (for the target language) and providing both text and audio output makes it much more user-friendly and effective.

Now, refine your own idea using these prompts. There are no right or wrong answers here. The aim is to practice thinking about how different streams of information can be woven together to create useful and interesting AI applications.

As you reflect on your idea, you might even start to anticipate some of the simple challenges discussed earlier in the course, like how to get these different types of data (aligning data from Chapter 2) or how a system might learn to connect them (integration techniques from Chapter 3). That's great! It shows you're already connecting the dots.

This kind of thinking is the first step in designing any AI system. By understanding the problem and the available information, engineers and designers can begin to sketch out solutions. Keep your notes; as you learn more about AI, you might revisit these ideas with new insights!