Multimodal Large Models
From text, images, audio, and video to mixed-content understanding
Suggested reading: about 20 min
Learning goal
Chapter keywords
| Keyword | Explanation | ESP32 engineering analogy |
|---|---|---|
| Modality encoder | A network that converts signals such as images or audio into representations usable by later computation. | Like the front end and driver dedicated to each sensor. |
| Cross-modal alignment | Making corresponding information from different modalities associable or jointly usable within a model. | Like defining a common contract for different interfaces. |
| Fusion | How a model combines text and other modality representations for a task, potentially at different network stages. | Like sending multiple sensor streams to a fusion task rather than simply joining raw voltage levels. |
| Temporal alignment | Preserving and relating the times of video frames, audio segments, and associated text. | Like using a shared clock to associate camera frames with microphone samples. |
Bridge from the previous chapter
The first sixteen chapters moved from neural networks to language models and deployment. This chapter extends input from text to images, audio, and video, applying Transformer, prefill, decode, and resource accounting to mixed content.
Historical development
Multimodal models build on several lines of research in vision representations, speech recognition, cross-modal alignment, and generative modeling. These milestones show how different inputs entered shared tasks; they do not imply that every model uses the same architecture.
Image captioning connects vision and language generation
Show and Tell combined image encoding with language generation to describe pictures, demonstrating an early end-to-end visual-to-text path.
Original Show and Tell paper ↗Vision Transformers treat image patches as a sequence
ViT divides an image into patches and processes them with a Transformer, demonstrating a sequential image representation with spatial information.
Original ViT paper ↗CLIP supervises vision with paired text
CLIP learns image–text correspondence with a contrastive objective, allowing textual descriptions to support image recognition and retrieval.
Original CLIP paper ↗Whisper expands audio–text supervision
Whisper trains speech recognition on large-scale audio–text pairs, demonstrating sequence modeling for transcription and related speech tasks.
Original Whisper paper ↗Visual instruction tuning connects images and dialogue
LLaVA connects a vision encoder and a language model, and uses visual instruction data to train image-and-text dialogue.
Original LLaVA paper ↗Temporal alignment enters long-video multimodal architectures
Mirasol3B models time-aligned audio-video and contextual modalities separately, showing why timing and fusion placement need explicit design.
Original Mirasol3B paper ↗Illustrated analogy
Multiple evidence streams enter a control room
A control room receives a written report, a photograph, and surveillance video with sound. An intake operator preserves each source, its order, and its time so the coordinator can answer: 'What was the light doing when the alarm sounded?' If only dialogue was transcribed, nobody can claim to have heard the alarm; if the camera supplies a frame only every few seconds, a flashing light may be missed.
Where the analogy stops: The model is not a conscious coordinator and may misunderstand any input. Architectures fuse representations differently; timestamps and citations still need independent verification.
Chapter walkthrough
Four inputs, four information structures
Text is converted by the model's tokenizer into token IDs and then embeddings; its order is normally one-dimensional. Images carry two-dimensional spatial and color information. They are often resized, normalized, divided into patches, and processed by a vision encoder for local and global features. Higher resolution or multiple crops generally increase the number of visual representations and compute. Audio is a waveform changing over time; it may be resampled, framed, and converted to a spectrogram before an audio encoder processes it. Some models instead use learned discrete audio codes. Video is a sequence of images with temporal relationships, potentially accompanied by a separate soundtrack. Frame sampling, timestamps, and synchronization determine what the model actually receives. Feature vectors need not be words from a text vocabulary, and an image need not first become a caption.
Learn representations, then connect modalities
A text language model commonly learns by predicting the next token in a sequence, acquiring patterns of expression and context. A vision encoder may learn from image tasks or paired images and text. CLIP, for example, uses a contrastive objective to bring matching images and descriptions together and distinguish mismatched pairs; this alignment does not make CLIP itself a complete conversational generator. Speech training can use paired audio and transcripts for recognition, or self-supervised learning to extract representations from unlabelled audio. Video training may use frames, soundtracks, subtitles, and time annotations to connect actions, sounds, and order. Inaccurate, shifted, or missing captions weaken that supervision. Each modality brings its own sampling rates, annotation costs, noise, and privacy requirements.
Connect the language model: alignment and instruction tuning
One common architecture pretrains vision or audio encoders and a language model separately, then trains a projector or adapter to map encoder outputs into vectors the language model can use. LLaVA demonstrates one route connecting a vision encoder, projector, and language model. Multimodal instruction tuning then uses images or audio together with questions and answers to improve evidence-based task behavior. Some training stages freeze components before unfreezing them; other systems use end-to-end joint training, cross-attention, or different arrangements. An autoregressive answer loss typically applies only at selected target-output positions, with the exact mask depending on the training recipe. Alignment enables representations to work together, while instruction tuning improves behavior; neither guarantees accurate perception or reasoning.
Follow each piece of evidence through mixed inference
Suppose a user submits a device photograph and a video with sound, asking: 'What is said in the video when the red light shown in the photo comes on?' The system first checks and decodes the media, preserving image order, sampled-frame timestamps, and audio time ranges. Text passes through a tokenizer; the photograph and video frames pass through a visual encoder, while the soundtrack is encoded as audio or transcribed separately. The system packages their representations with modality markers, positions, and available time information according to the model's protocol. Fusion may concatenate visual and audio vectors with the text sequence, use cross-attention, or connect separate stages. Not every system supports all four native inputs. The model processes context in prefill and generates text step by step in decode. The answer depends on the actual input, sampling, and model capability: correct localization of the light or recognition of speech is not guaranteed.
Audio and video: time is part of the evidence
Speech recognition conveniently turns sound into text, but can discard intonation, overlapping speakers, music, alarms, and other nonverbal evidence; it can also mishear or fabricate segments. Native audio input can preserve more acoustic information, but remains limited by its encoding and training. Sparse video sampling can miss brief changes, and incorrect frame order or shifted timestamps can reverse what 'before' and 'after' mean. Audio-video synchronization needs a shared time reference and transcript segments tied to time ranges. A streaming system must also handle chunk boundaries, buffering delay, and late frames: it cannot use information that has not arrived. An offline system, by contrast, may process the entire clip before answering. Compare both paths using examples with known timing.
Outputs and resources: text, speech, and real-time interaction
A text response is usually generated token by token and may reuse a KV cache. Speech output may first generate text and send it to a separate text-to-speech system, or a multimodal model may predict audio codes that a decoder reconstructs as a waveform. The intermediate text in the first route is easier to inspect, but acoustic detail may be lost. Available voices, latency, and control in the second route depend on the particular model. Real-time interaction requires continuous audio input, endpoint detection, chunked generation, interruption, and resource reclamation. Multiple high-resolution images, long audio, and many video frames increase encoding work and context usage, potentially raising time to first token, cache size, memory, and energy. Compression differs across models, so there is no universal fixed token count per image.
Validate with counterexamples and traceable evidence
Evaluation should cover small text, occlusion, low light, accents, noise, overlapping speakers, brief actions, and shifted audio-video timing. Test text alone, images or audio alone, and mixed input separately. Record the model version, preprocessing, sampling policy, modalities actually supplied, time to first token, and peak memory so that changes in quality and performance can be explained. Ask for verifiable frame times or original audio segments, but check the citations too: a model can guess nonexistent details from language priors. When evidence is absent, the system should explicitly express uncertainty. For device control, enforce deterministic authorization, value ranges, and safety interlocks; never treat a generated response as an execution command without those checks.
Interactive process
One mixed image, text, audio, and video question
Track the source, time, and encoding of each input, then see how representations enter context and generate an answer; fusion varies by model.
Receive request · text + image + audio + video
Check supported inputs, files, and order
A system may not support every native modality
Loop / return condition: Autoregressive text generation repeats the final stage until stopping; streaming systems also receive new chunks and handle interruptions, with caching and synchronization depending on the implementation.
View the complete static diagram
text → tokenizer → text vectors ──────────────┐ image → patches / vision encoder → features ──┤ audio → frames / audio encoder → features ────┼→ fusion + context → inference → text / speech video → sampled frames + soundtrack + times ─┘
Code or command example
# Illustrative only: interfaces, fusion, and outputs vary by model
parts = [encode_text(question), encode_image(photo),
encode_audio(audio), encode_video_frames(video, timestamps=True)]
context = assemble_with_order_and_timestamps(parts)
state = multimodal_model.prefill(context)
while not finished:
token, state = multimodal_model.decode(state)
emit(token) # Speech also needs a suitable audio generation/decoding pathHands-on lab
Lab notes and export
Engineering pitfall
Knowledge check
Answer all three multiple-choice questions, then submit. Answers stay hidden until submission.
Discuss this chapter on GitHub
Sign in with GitHub to ask a question, share measurements, or compare implementations. Comments are stored in this course's GitHub Discussions.
The embedded comments need JavaScript. You can also open the GitHub discussion area directly: Open GitHub Discussions ↗
Further reading
Bridge to the next chapter
After the course, choose a real scenario and define acceptance using traceable audio-video evidence, modality ablations, failure examples, and an edge resource budget.