Lesson 94: Multimodal Models

Multimodal models are AI systems that can understand and work with more than one type of data at the same time. Instead of using only images or only text, these models combine multiple data sources such as images, text, audio, and video.

By learning from different modalities together, multimodal models gain a richer and more human-like understanding of information.

Real-World Connection

Multimodal models power modern applications such as image captioning, visual question answering, voice assistants with screens, content moderation, and AI systems that can see and read at the same time.

For example, when an AI describes what is happening in a photo or answers questions about an image using text, it is using multimodal learning.

What Does “Multimodal” Mean?

A modality refers to a type of data. Common modalities include:

• Images
• Text
• Audio
• Video
• Sensor data

A multimodal model processes two or more of these modalities together instead of independently.

Why Multimodal Models Are Important

Real-world information is rarely limited to a single data type. Humans naturally combine vision, language, and sound to understand the world.

Multimodal models aim to replicate this ability by learning relationships across different data sources.

• Improves understanding accuracy
• Handles complex tasks better
• Enables richer AI applications

How Multimodal Models Work

Most multimodal models follow a similar structure:

• Each modality is processed by its own encoder
• Encoded features are combined (fusion)
• The fused representation is used for prediction

Fusion can happen early (raw data), middle (features), or late (outputs).

Common Multimodal Architectures

• Image + Text: CNN or ViT with language models
• Audio + Text: Speech models with NLP
• Vision + Language Transformers: Joint attention models

Simple Multimodal Example (Image + Text)

Below is a simplified example showing how image and text features can be combined in a neural network.

import tensorflow as tf
from tensorflow.keras import layers

# Image input branch
image_input = layers.Input(shape=(224, 224, 3))
x_img = layers.Conv2D(32, 3, activation='relu')(image_input)
x_img = layers.GlobalAveragePooling2D()(x_img)

# Text input branch
text_input = layers.Input(shape=(100,))
x_txt = layers.Dense(64, activation='relu')(text_input)

# Fusion
combined = layers.Concatenate()([x_img, x_txt])
output = layers.Dense(1, activation='sigmoid')(combined)

model = tf.keras.Model(
    inputs=[image_input, text_input],
    outputs=output
)
  

What This Code Is Doing

The image branch extracts visual features, while the text branch processes textual input. These features are then merged into a single representation.

The combined features allow the model to make decisions based on both image and text information.

Understanding the Output

The model outputs a prediction that depends on both inputs. Changing either the image or the text can affect the final result.

This demonstrates how multimodal systems integrate information across modalities.

Popular Multimodal Use Cases

• Image caption generation
• Visual question answering
• Content moderation
• Assistive technologies

Challenges in Multimodal Learning

• Data alignment across modalities
• High computational cost
• Complex training pipelines

Despite these challenges, multimodal models are becoming increasingly important in advanced AI systems.

Practice Questions

Quick Quiz

Coming up next: Computer Vision Use Cases — applying vision techniques to real-world problems.