Edge AI · A 17-Day Engineering Path
Course home 简体中文
DAY 17

Multimodal Large Models

From text, images, audio, and video to mixed-content understanding

Suggested reading: about 20 min

Learning goal

Explain how different modalities are represented, trained, and supplied to a model; trace a mixed text, image, audio, and video inference request; and compare text responses, speech output, real-time operation, and reliability limits.

Chapter keywords

KeywordExplanationESP32 engineering analogy
Modality encoderA network that converts signals such as images or audio into representations usable by later computation.Like the front end and driver dedicated to each sensor.
Cross-modal alignmentMaking corresponding information from different modalities associable or jointly usable within a model.Like defining a common contract for different interfaces.
FusionHow a model combines text and other modality representations for a task, potentially at different network stages.Like sending multiple sensor streams to a fusion task rather than simply joining raw voltage levels.
Temporal alignmentPreserving and relating the times of video frames, audio segments, and associated text.Like using a shared clock to associate camera frames with microphone samples.

Bridge from the previous chapter

The first sixteen chapters moved from neural networks to language models and deployment. This chapter extends input from text to images, audio, and video, applying Transformer, prefill, decode, and resource accounting to mixed content.

HOW WE GOT HERE

Historical development

Multimodal models build on several lines of research in vision representations, speech recognition, cross-modal alignment, and generative modeling. These milestones show how different inputs entered shared tasks; they do not imply that every model uses the same architecture.

2014

Image captioning connects vision and language generation

Show and Tell combined image encoding with language generation to describe pictures, demonstrating an early end-to-end visual-to-text path.

Original Show and Tell paper ↗
2020

Vision Transformers treat image patches as a sequence

ViT divides an image into patches and processes them with a Transformer, demonstrating a sequential image representation with spatial information.

Original ViT paper ↗
2021

CLIP supervises vision with paired text

CLIP learns image–text correspondence with a contrastive objective, allowing textual descriptions to support image recognition and retrieval.

Original CLIP paper ↗
2022

Whisper expands audio–text supervision

Whisper trains speech recognition on large-scale audio–text pairs, demonstrating sequence modeling for transcription and related speech tasks.

Original Whisper paper ↗
2023

Visual instruction tuning connects images and dialogue

LLaVA connects a vision encoder and a language model, and uses visual instruction data to train image-and-text dialogue.

Original LLaVA paper ↗
2024

Temporal alignment enters long-video multimodal architectures

Mirasol3B models time-aligned audio-video and contextual modalities separately, showing why timing and fusion placement need explicit design.

Original Mirasol3B paper ↗
Why it still matters today: These studies address representation, alignment, speech, or temporal fusion in different ways. A product must establish its supported modalities and input protocol, then evaluate target data rather than generalizing one paper's architecture to every system.
BUILD INTUITION FROM A FAMILIAR SYSTEM

Illustrated analogy

Multiple evidence streams enter a control room

A control room receives a written report, a photograph, and surveillance video with sound. An intake operator preserves each source, its order, and its time so the coordinator can answer: 'What was the light doing when the alarm sounded?' If only dialogue was transcribed, nobody can claim to have heard the alarm; if the camera supplies a frame only every few seconds, a flashing light may be missed.

Written report Tokenizer and text context
Site photograph Image preprocessing and visual features
Surveillance audio Audio encoding or lossy speech transcription
Shared event timeline Video frames, audio segments, and temporal alignment

Where the analogy stops: The model is not a conscious coordinator and may misunderstand any input. Architectures fuse representations differently; timestamps and citations still need independent verification.

Chapter walkthrough

Four inputs, four information structures

Text is converted by the model's tokenizer into token IDs and then embeddings; its order is normally one-dimensional. Images carry two-dimensional spatial and color information. They are often resized, normalized, divided into patches, and processed by a vision encoder for local and global features. Higher resolution or multiple crops generally increase the number of visual representations and compute. Audio is a waveform changing over time; it may be resampled, framed, and converted to a spectrogram before an audio encoder processes it. Some models instead use learned discrete audio codes. Video is a sequence of images with temporal relationships, potentially accompanied by a separate soundtrack. Frame sampling, timestamps, and synchronization determine what the model actually receives. Feature vectors need not be words from a text vocabulary, and an image need not first become a caption.

Learn representations, then connect modalities

A text language model commonly learns by predicting the next token in a sequence, acquiring patterns of expression and context. A vision encoder may learn from image tasks or paired images and text. CLIP, for example, uses a contrastive objective to bring matching images and descriptions together and distinguish mismatched pairs; this alignment does not make CLIP itself a complete conversational generator. Speech training can use paired audio and transcripts for recognition, or self-supervised learning to extract representations from unlabelled audio. Video training may use frames, soundtracks, subtitles, and time annotations to connect actions, sounds, and order. Inaccurate, shifted, or missing captions weaken that supervision. Each modality brings its own sampling rates, annotation costs, noise, and privacy requirements.

Connect the language model: alignment and instruction tuning

One common architecture pretrains vision or audio encoders and a language model separately, then trains a projector or adapter to map encoder outputs into vectors the language model can use. LLaVA demonstrates one route connecting a vision encoder, projector, and language model. Multimodal instruction tuning then uses images or audio together with questions and answers to improve evidence-based task behavior. Some training stages freeze components before unfreezing them; other systems use end-to-end joint training, cross-attention, or different arrangements. An autoregressive answer loss typically applies only at selected target-output positions, with the exact mask depending on the training recipe. Alignment enables representations to work together, while instruction tuning improves behavior; neither guarantees accurate perception or reasoning.

Follow each piece of evidence through mixed inference

Suppose a user submits a device photograph and a video with sound, asking: 'What is said in the video when the red light shown in the photo comes on?' The system first checks and decodes the media, preserving image order, sampled-frame timestamps, and audio time ranges. Text passes through a tokenizer; the photograph and video frames pass through a visual encoder, while the soundtrack is encoded as audio or transcribed separately. The system packages their representations with modality markers, positions, and available time information according to the model's protocol. Fusion may concatenate visual and audio vectors with the text sequence, use cross-attention, or connect separate stages. Not every system supports all four native inputs. The model processes context in prefill and generates text step by step in decode. The answer depends on the actual input, sampling, and model capability: correct localization of the light or recognition of speech is not guaranteed.

Audio and video: time is part of the evidence

Speech recognition conveniently turns sound into text, but can discard intonation, overlapping speakers, music, alarms, and other nonverbal evidence; it can also mishear or fabricate segments. Native audio input can preserve more acoustic information, but remains limited by its encoding and training. Sparse video sampling can miss brief changes, and incorrect frame order or shifted timestamps can reverse what 'before' and 'after' mean. Audio-video synchronization needs a shared time reference and transcript segments tied to time ranges. A streaming system must also handle chunk boundaries, buffering delay, and late frames: it cannot use information that has not arrived. An offline system, by contrast, may process the entire clip before answering. Compare both paths using examples with known timing.

Outputs and resources: text, speech, and real-time interaction

A text response is usually generated token by token and may reuse a KV cache. Speech output may first generate text and send it to a separate text-to-speech system, or a multimodal model may predict audio codes that a decoder reconstructs as a waveform. The intermediate text in the first route is easier to inspect, but acoustic detail may be lost. Available voices, latency, and control in the second route depend on the particular model. Real-time interaction requires continuous audio input, endpoint detection, chunked generation, interruption, and resource reclamation. Multiple high-resolution images, long audio, and many video frames increase encoding work and context usage, potentially raising time to first token, cache size, memory, and energy. Compression differs across models, so there is no universal fixed token count per image.

Validate with counterexamples and traceable evidence

Evaluation should cover small text, occlusion, low light, accents, noise, overlapping speakers, brief actions, and shifted audio-video timing. Test text alone, images or audio alone, and mixed input separately. Record the model version, preprocessing, sampling policy, modalities actually supplied, time to first token, and peak memory so that changes in quality and performance can be explained. Ask for verifiable frame times or original audio segments, but check the citations too: a model can guess nonexistent details from language priors. When evidence is absent, the system should explicitly express uncertainty. For device control, enforce deterministic authorization, value ranges, and safety interlocks; never treat a generated response as an execution command without those checks.

WATCH THE DATA MOVE

Interactive process

One mixed image, text, audio, and video question

Track the source, time, and encoding of each input, then see how representations enter context and generate an answer; fusion varies by model.

Step 1 / 6

Receive request · text + image + audio + video

Check supported inputs, files, and order

Watch for

A system may not support every native modality

Loop / return condition: Autoregressive text generation repeats the final stage until stopping; streaming systems also receive new chunks and handle interruptions, with caching and synchronization depending on the implementation.

View the complete static diagram
text → tokenizer → text vectors ──────────────┐
image → patches / vision encoder → features ──┤
audio → frames / audio encoder → features ────┼→ fusion + context → inference → text / speech
video → sampled frames + soundtrack + times ─┘

Code or command example

# Illustrative only: interfaces, fusion, and outputs vary by model
parts = [encode_text(question), encode_image(photo),
         encode_audio(audio), encode_video_frames(video, timestamps=True)]
context = assemble_with_order_and_timestamps(parts)
state = multimodal_model.prefill(context)
while not finished:
    token, state = multimodal_model.decode(state)
    emit(token)  # Speech also needs a suitable audio generation/decoding path

Hands-on lab

Choose a short video with visuals, dialogue, and environmental sounds, and ask: 'What happened before and after the alarm?' First record a transcript, sampled frames, audio segments, and timestamps separately; then draw how each enters the model. Compare text only, text plus frames, and text plus frames and audio. Record answers, missing evidence, errors, and inference latency. Change the sampling interval and check whether a brief event disappears. Confirm that the model actually accepts native audio: a transcript is not evidence that it heard an alarm. Finally budget privacy permissions, resolution, audio duration, and peak memory.
Lab notes and export

Engineering pitfall

Avoid this mistake: Assuming all multimodal models first convert every image and sound to text, or that every video frame and sound was observed. Actual input support, sampling, and fusion depend on the model and system; verify time alignment and missing evidence.

Knowledge check

Answer all three multiple-choice questions, then submit. Answers stay hidden until submission.

1. What is one common way to connect images to a language model?
2. Why preserve sampled-frame timestamps and audio-video synchronization in video question answering?
3. Which statement best describes multimodal training and inference?

LEARN TOGETHER

Discuss this chapter on GitHub

Sign in with GitHub to ask a question, share measurements, or compare implementations. Comments are stored in this course's GitHub Discussions.

The embedded comments need JavaScript. You can also open the GitHub discussion area directly: Open GitHub Discussions ↗

Further reading

Bridge to the next chapter

After the course, choose a real scenario and define acceptance using traceable audio-video evidence, modality ablations, failure examples, and an edge resource budget.