Multimodal AI: When Models See and Hear
Updated: Oct 8
UIT University 365 Institute of Technology
Series AI Frontiers | Level Basic (Free)
Duration 25 minutes | Access Free
IT Engineering, AI and Applied AI, Data Science, Software Development, Digital Transformation
UNOP Sound (University 365 Neuroscience Oriented Pedagogy)
Take five minutes to prepare your brain. Play the isochronous tone track (40Hz gamma frequency) with your eyes closed. Gamma-frequency tones before a learning session raise attention and make the material easier to absorb.
[Audio player: UNOP Pre-Lecture Isochrone (40Hz, 5 minutes)]
In this Lecture
You Asked About a Chart and a Recording
You send a model a photograph of a revenue chart and a 90-second audio clip of the meeting where that chart was presented. You ask one question: "Did the speaker's explanation match what the chart actually shows?"
A text-only model cannot answer that at all. A pipelined system will try: a vision model describes the chart, a speech recognizer transcribes the audio, and a language model compares two pieces of text. That works for easy cases. It fails the moment the answer depends on something that survives only in the original signal, such as the speaker pausing before a number, or a chart axis that does not start at zero.
Multimodal models exist to close that gap. They process images, audio, and video alongside text, and the strongest of them reason across the modalities inside one network rather than across a chain of models. This lecture shows you how that works mechanically, what each architecture family buys and costs, and where multimodal systems still fail. By the end you will be able to choose a multimodal approach deliberately instead of by vendor habit.
What Multimodal Actually Means
The word multimodal is used loosely, and the looseness causes bad engineering decisions. Separate the four senses before you specify anything.
Input-multimodal. The model accepts more than one input type and returns text. Almost any capable language model today can be wrapped in an image preprocessor and called input-multimodal. This is the weakest sense and the one most vendor material uses.
Output-multimodal. The model generates more than one output type, such as text and images. Most systems that do this are a text model with separate generative heads attached, not one network that learned both.
Unified-token multimodal. Every modality is converted into tokens in one shared vocabulary and pushed through one transformer, which then handles both understanding and generation. This is a real architectural commitment, not a wrapper.
Native cross-modal reasoning. The model answers a question that cannot be answered from any single modality alone. Your chart-plus-audio question is the test case. This is what practitioners usually mean by multimodal, and it is the sense that matters.
The test that separates them
Ask a candidate system the chart-and-recording question and inspect the output. If the answer traces back to a text description of the chart, you are using a pipeline. If the model can point at a visual detail and a spoken detail in the same sentence, and its confidence tracks how clearly each was captured, you have cross-modal reasoning. The distinction is not academic: it decides whether your error rate stays flat or rises silently as inputs get harder.
Why this became the default
Three pressures pushed multimodality from a feature to a baseline. Documents and screenshots became the most common thing people want a model to read. Voice became the most natural interface. And the video that organizations already record turned out to be the largest untapped input they own. A model that cannot take those inputs forces you to build lossy conversion steps before the model even starts working.
How a Model Reads an Image
A transformer cannot look at pixels. It consumes sequences of vectors. So the first job of any vision-capable model is to turn an image into something sequence-shaped. That conversion is where most of your cost decisions live.
Patches, then vectors
An image is divided into a grid of small squares, typically 16 by 16 or 32 by 32 pixels. Each patch is passed through a vision encoder, usually a Vision Transformer, which converts it into a vector. Those vectors are then projected into the same space the text tokens occupy, so the attention mechanism can weigh an image patch and a word in the same operation.
Nothing about this is metaphorical. The model genuinely treats the image as a long sequence of tokens that it reads next to your prompt.
What that costs
Image input is billed in tokens, and the arithmetic is worth knowing because it changes how you design a pipeline. On the tile-based schemes used by the GPT-4o and GPT-4.1 families, a low-detail image costs a flat 85 tokens. A high-detail image costs 85 base tokens plus 170 tokens for each 512 pixel square after the image is scaled to fit. A 1024 by 1024 image therefore costs 85 plus 170 times 4, which is 765 tokens. A 2048 by 4096 image costs 1105 tokens.
Two practical consequences follow. First, image tokens are not free next to a 750-token text prompt: one high-detail image can more than double the request. Second, resolution is a knob you control. Resize an image to the smallest size that still preserves the detail you need, and you cut cost immediately without changing the model.
The layout-aware alternative
Some models process a document page as an image patch grid rather than extracting text first. You keep table structure, column alignment, and the visual relationship between a label and its value. For invoices, forms, and dense reports, layout-aware reading is usually more accurate than an OCR-then-reason chain, because OCR throws away exactly the spatial information that made the page readable to a human.
Three Fusion Architectures
The 2024 to 2026 literature converged on three patterns, and they differ in one place: where in the network the modalities meet. That single decision cascades into cost, latency, and which cross-modal tasks the model can learn at all.
Late fusion
Train a separate text encoder and a separate vision encoder, project both into a shared embedding space, and compare vectors by cosine similarity. This is the CLIP lineage and the ancestor of the LLaVA-style small vision-language models.
Late fusion is cheap, well understood, and unbeatable for retrieval. Use it when you need to find an image by a text description, or match a caption to a picture at scale. What it cannot do is reason across modalities. It can retrieve across them, which is a different capability.
Adapter tokens
Train or take a strong text model, then attach modality encoders whose outputs are projected into the text model's token space. The text stack stays largely frozen. This became the dominant production pattern because it inherits text reasoning for the price of an adapter and degrades gracefully when an encoder fails.
The trade-off is structural. The model never learns a genuinely shared abstraction: it translates vision and audio into text-shaped concepts, then reasons in text. Information that does not survive translation is gone.
Native, or early fusion
Replace the text-only backbone with a unified transformer that ingests tokens from every modality from the first layer. Images may be quantized into discrete tokens and mixed with text tokens, and the same network learns to caption and to generate. Cross-attention operates on the interleaved sequence throughout the depth of the model.
Native designs are harder to train and more expensive to run at high video resolution, since every extra frame or second of audio adds tokens. What they buy is the capability that gives this lecture its name: cross-modal reasoning that does not route through a single bottleneck modality.
The honest summary
A pipeline of strong single-modality models beats a weak native model on simple tasks, and it is faster to ship. Native and adapter designs win when the answer genuinely depends on two modalities at once. Choose by the task, not by the architecture label.
How Audio and Video Become Tokens
Audio and video look harder than images, and mechanically they are the same trick applied along a time axis.
Audio becomes frames
A waveform is sliced into short windows, typically tens of milliseconds. Each window is converted to a spectrogram slice or an acoustic feature vector, then encoded into embeddings that land in the shared space. Some designs project frames directly into the token space with a linear layer and drop the separate audio encoder entirely; others run a conformer stack over log-mel features and project the output.
The important property is what the representation preserves. A transcription keeps words and throws away tone, pace, emphasis, and background. An acoustic embedding can keep them. That is why a native audio model can react to an interruption mid-sentence, while a transcribe-then-reason pipeline cannot.
On latency: OpenAI's GPT-4o system card reports audio responses in as little as 232 milliseconds, with an average of 320 milliseconds, which is in the range of human conversational turn-taking. That number is an architecture result, not a tuning result. Audio that never leaves the model's own network avoids the transcribe-then-call round trip.
Video is sampled frames
Most multimodal systems do not watch video continuously. They sample frames at a fixed rate, commonly between one and five frames per second, and process that frame sequence. This has three consequences you should design around:
Fast motion is invisible. A 0.2 second action can fall entirely between two sampled frames.
Slow changes are visible. A gauge rising over 30 seconds is captured, if the sample rate is high enough.
Token count scales with duration. An hour of footage at even one frame per second is thousands of image tokens before any audio.
Long context windows make video practical by letting many frames plus audio sit in one sequence. They do not make it free, and they do not turn sampling into continuous perception.
Interleaving is the point
In a native model, the tokens are interleaved into a single sequence: frame tokens, audio tokens, and text tokens in reading order. Attention then operates across all of them. That is what allows an answer to reference the slide that was on screen at the moment a particular sentence was spoken.
Cross-Modal Reasoning: The Task That Changes Everything
Here is the concrete difference, on a task you can run yourself today.
Take a bar chart where the y-axis starts at 95 rather than zero, so a change from 100 to 140 looks like a vertical cliff. Put the caption below it: "sales increased 40 percent in the third quarter."
A pipelined system converts the chart to text: a list of four bar values. The language model reads those values, computes a 40 percent increase, and reports that the caption is accurate. It is technically correct about the numbers and completely wrong about the impression the chart creates.
A model that processes the chart pixels and the caption text in the same attention layers can notice the axis origin, compare the visual slope against the claimed magnitude, and flag the mismatch. The finding depends on information that exists only in the relationship between what the image shows and what the text claims.
Where this pays off
Four task shapes consistently separate native multimodal from pipelines:
Cross-validation. Does the document say what the spreadsheet shows? Does the label match the contents?
Temporal grounding. What was on screen when the speaker said that? What changed between minute 3 and minute 7?
Layout-dependent extraction. Tables, forms, and invoices where the position of a value carries meaning.
Interruptible interaction. Live conversation where tone and timing change the correct response.
If your use case is none of these, a pipeline is often the better engineering choice: it is cheaper, easier to debug, and you can swap components independently.
What Multimodal Models Still Get Wrong
Enthusiasm for these systems collapses quickly in production if you do not know the failure modes. These are the four that matter most.
Audio degrades fastest
Text understanding is near human level for many tasks. Vision is strong on clean images and weaker on dense diagrams, small text, and low-contrast screenshots. Audio is the least mature modality by a wide margin. Background noise, overlapping speakers, accents outside the training distribution, and non-speech audio all degrade accuracy quickly. In real call-center and meeting conditions, plan for double-digit error rates rather than single-digit ones, and preprocess aggressively: reduce noise, separate speakers, and remove silence before the audio ever reaches the model.
Sampling masquerades as understanding
Because video is sampled, a system can produce a fluent summary while having missed the event you cared about. When the task is temporal, control the sample rate yourself and state the rate in your evaluation. Never infer that a model "watched" something continuously just because the summary reads well.
Small text in images is fragile
A screenshot scaled down to save image tokens can lose the very text you need. This is a direct tension with the cost arithmetic above: shrinking an image to cut tokens also cuts legibility. Resolve it by cropping to the region that matters instead of downscaling the whole frame.
Confidence does not track accuracy across modalities
Models tend to sound equally certain about a clean transcript, a noisy recording, and a misread chart. Build your own checks. Where a decision carries cost, verify the extracted value against a second path, such as a table lookup, rather than trusting a fluent answer.
Cost, Latency, and Context: The Engineering Constraints
Multimodal capability is easy to demo and expensive to run at scale. Four levers control almost all of it.
Resolution and detail level
Image cost is driven by dimensions and detail setting, and the difference is large: 85 tokens for a low-detail image against 765 for the same 1024 by 1024 image at high detail. Choose the lowest detail that still answers your question, and crop before you send.
Frame rate and duration
Video token count scales with frame count, which scales with duration times sample rate. Halving your sample rate halves the video tokens. Test whether the task actually needs the higher rate before you pay for it.
Context budget
Multimodal inputs consume the context window that your instructions and retrieved documents also need. A single hour of video can crowd out the entire rest of a request. Budget tokens per modality explicitly rather than discovering the overflow in production.
Latency
A pipeline adds a round trip per stage: transcribe, then reason, then respond. A native model answers inside one forward pass, which is why conversational audio latency can sit near 320 milliseconds instead of seconds. If your product is interactive, latency is an architecture decision, not a hosting decision.
The evaluation you actually need
Build a fixed set of real inputs before you tune anything: your own charts, your own noisy recordings, your own screenshots. Score accuracy, latency, and cost per item. Vendor demos use clean inputs. Your inputs are not clean, and fifty representative examples will tell you more than any leaderboard.
The Practical Decision Framework
Work through four questions in order. The answers determine the architecture, and skipping a question is how teams end up paying for capability they cannot use.
1. Inventory the data you actually hold. Not the data you could collect. If your inputs are text and clean PDFs, you have a document problem, not a multimodal problem, and a layout-aware extraction model is enough.
2. Ask whether the answer crosses modalities. If every question can be answered from one modality at a time, build the pipeline. It is cheaper, more debuggable, and independently swappable. If the answer needs two modalities at once, a native model is the correct tool.
3. Measure on representative samples. Accuracy, latency, and cost per item, on your inputs, with the sample rate and resolution you can afford in production.
4. Set the resolution and frame rate from the failure mode, not the budget. Find the point where quality falls off. Crop instead of downscale. Cut duration instead of frame rate when the task is about a moment, and cut frame rate instead of duration when the task is about a trend.
A worked choice
Say you are building a support tool that ingests a screenshot plus the customer's written description and routes the ticket. Every routing decision can be made from the screenshot's text alone, so you have a single-modality problem. Use an image-capable model at moderate resolution, extract the visible text and error codes, and route on that. Reserve a native multimodal model for the minority of tickets where the customer's description and the screenshot disagree, because those are the cases where cross-validation earns its cost.
That is the discipline in one line: match the architecture to the task shape, and pay for cross-modal reasoning only where the answer requires it.
The Latency Budget You Have to Fill In
Interactive multimodal work is judged on latency before it is judged on accuracy, and latency is the sum of parts you can list. Write the list before you choose an architecture.
Where the milliseconds go
Take a voice assistant that answers a question about a chart on the user's screen. The pipeline path has four stages, each with its own cost:
Stage | Pipeline | Native multimodal |
Capture and transport | 30 to 80 ms | 30 to 80 ms |
Speech recognition | 150 to 400 ms | inside the model |
Image fetch, resize, encode | 60 to 150 ms | 60 to 150 ms |
Reasoning over text | 300 to 900 ms | 300 to 900 ms |
Speech synthesis | 100 to 300 ms | inside the model, streamed |
Round trips between services | 2 x 20 to 60 ms | none |
Add the pipeline column and you land between 700 milliseconds and two seconds before any queueing. The published figure for a native audio model is a 232 millisecond best case and a 320 millisecond average, because the whole exchange runs inside one forward pass. The gap is not a vendor claim you have to take on faith: it is the four round trips and the separate speech component disappearing from the list.
Three ways to buy latency back
Stream. A first token in 300 milliseconds feels faster than a whole reply in 900, even though the total work is the same.
Cut the input before it reaches the model. Crop an image to the region the question is about and trim audio to the minute that matters. Input size drives both token count and encode time.
Start the slow part first. If the answer needs a transcript and a chart, request both in parallel. Two 400 millisecond calls in parallel take 400 milliseconds, not 800.
For a conversational product the usual target is under 800 milliseconds to first audio and under three seconds to a complete answer, measured at the 95th percentile. Measure it on your own inputs, because queueing and provider variance dominate the last mile and neither appears in a vendor table.
Building the Fifty-Item Test Set
A multimodal system cannot be evaluated on a leaderboard, because the leaderboard does not use your charts, your accent distribution, or your screenshots. Build the set yourself, before you tune anything.
What goes in the fifty
Category | Count | What it tests |
Clean text in an image | 10 | The floor. A competent model should be near perfect here. |
Dense layout with a table or a form | 10 | Whether structure survives the trip through the model. |
Low-contrast or small-text screenshot | 10 | Where downscaling for token cost starts to cost accuracy. |
Noisy or overlapped audio | 10 | The least mature modality, measured rather than assumed. |
Cross-modal questions | 10 | The cases where the answer needs two modalities at once. |
Keep the last row small on purpose. Most production questions use one modality at a time, and a set that is mostly cross-modal will push you toward an architecture you do not need.
How to score it
Score three things per item. Correct or not, by a written rule for what counts as correct: write the rule before you look at any output, or you will grade every run generously. Latency, in milliseconds. Cost, in tokens or dollars, so the routing decision has a number behind it. Record the sample rate and resolution alongside each item, or the accuracy figure is not reproducible.
Three readings change a decision. Accuracy flat across the first three rows but falling on the noisy-audio row means you have an audio problem, and a better vision model will not fix it: preprocess the audio instead. Accuracy falling only on the small-text row means you are over-shrinking images to save tokens, so crop the region of interest rather than downscaling the whole frame. A cross-modal row near chance while every single-modality row is strong means your pipeline is converting the inputs to text before reasoning, and that task needs a different system rather than a better prompt.
Re-run the set on every change to a model version, a detail setting, or a preprocessing step. Fifty items at a few cents each is cheap; discovering a silent accuracy loss from your users is not.
Where Multimodal Data Lives in Your Store
A multimodal feature is not a model decision alone. It is a storage and indexing decision too, and that part is usually underestimated because a demo never has more than three files in it.
Four things you now have to store
Asset | What you keep | Why it is not optional |
The original bytes | The image, audio or video file itself | You re-run extraction after a model change |
A derived representation | Extracted text, a transcript, a structured record | Cheap to search, and what most queries hit |
An embedding | A vector per chunk, per frame set, or per page | What makes similarity search possible |
Provenance | Source, capture date, page or timestamp offset, model version | Without it you cannot cite a result or invalidate a bad extraction |
Provenance is the one teams add last and need first. When a model produces a wrong answer from a document, the only way to find out why is to know which page, which frame, and which extraction version fed it.
Chunking is different for each modality
Text chunking has a settled habit: a few hundred tokens with some overlap. The other modalities do not behave the same way.
Images. One image is usually one unit, but a dense page with a table is not. Split a page at structural boundaries: a table, a signature block, a chart with its caption. A caption belongs with its image, because a retrieved image without its caption is often unusable.
Audio. A timestamped window, typically 20 to 60 seconds, cut on natural pauses where possible. Store the start and end offsets with the embedding, because a transcript chunk without a timestamp cannot be played back, and playback is how a human checks the answer.
Video. Two representations, not one: a transcript keyed to the audio, and sampled frames keyed to their timestamps. A query about what was said hits the first; a query about what was shown hits the second. Keeping only the transcript throws away the visual half of the video.
The index cost nobody budgets
Embeddings are cheap per item and not cheap per corpus. One hour of recording transcribed and chunked at 30 seconds is about 120 text chunks. The same hour sampled at one frame per second, with each frame embedded, is 3,600 items. The visual index can be thirty times the size of the transcript index for the same hour of content.
Two consequences follow. The frame sample rate is an indexing decision, not a generation decision alone: the rate you choose at indexing time is the rate you can search at later, and lowering it at query time cannot bring back a frame you never stored. And index the transcript of everything and the frames of what matters, because frame-level indexing for a whole archive is often ten to fifty times the cost of transcript-level indexing, for a capability only some queries use.
Permission is harder here
Text retrieval has one permission model: which rows can this user read. Multimodal retrieval usually has a second one, because the same recording contains several people and the same photograph contains faces and documents that may not belong to the caller. Three rules keep this tractable:
Store permissions with the chunk, not with the file. A chunk inherits the narrowest permission of anything visible or audible in its span.
Redact at ingestion, not at display. If a region or a voice must not be shown to a given audience, remove it from the derived representation the retriever searches. A display-time filter is one bug away from a leak.
Keep the raw asset behind a second check. The embedding and the transcript are what the model reads. The original is what a human opens, and it should require its own authorisation step.
The Privacy Surface Multimodal Adds
Sending an image or a recording to a model sends more than a file. It sends whatever is in the frame and in the room, and that is a different category of data from the text you were sending before.
What each modality carries that text does not
Images carry faces, badges, whiteboards, screens with other tabs open, and the contents of a desk. A screenshot of a support ticket typically contains a customer name and an account number.
Audio carries voices, which are biometric identifiers in several jurisdictions, plus whatever conversation happened near the microphone before and after the part you meant to record.
Video carries all of the above plus location, timing, and who was in the room with whom.
This is not a reason to avoid multimodal work. It is a reason to decide the retention and redaction rule at the same time you decide the architecture, because both are cheap at design time and expensive afterwards.
Four decisions to make before the first upload
Where does the raw asset live? In your own store, in the model provider's storage, or not at all. A provider's file-upload API usually keeps the file for a retention window, and that window is often a configuration you can shorten.
What gets redacted before it leaves? A crop to the region the question is about removes most of the incidental content. Blanking faces and account numbers costs a fraction of a second.
How long is the derived representation kept? Transcripts and extracted text are searchable, and therefore more useful and more dangerous than the source. Apply the retention rule to the derived record as well as to the original file.
Who can retrieve it? Covered above, and it belongs in the same design pass, not a later project.
When to keep the work local
A local system accepts an image or a recording without any of it leaving the machine. For a regulated workflow, such as medical images, legal documents, or personnel recordings, that property is often the deciding argument rather than a cost argument.
The trade is the one you already know from the rest of this lecture: a local model is weaker than a hosted frontier model on hard cross-modal questions, and you own the operations. Run the high-volume, low-stakes, private work locally, such as extracting a field from a form, and send only the cases that genuinely need frontier cross-modal reasoning, with the input cropped to the region that matters.
The check to run on your own feature
Take one real input from each modality your product accepts and answer four questions in writing: what personal data is visible in it, where the file is stored, how long each derived record is kept, and who can query it. If any answer is "we would have to check", that is the work to do next.
Feynman Summary: Explain It Like You Are 12
Imagine you have three friends. One only reads, one only looks at pictures, and one only listens.
If you want to know whether a picture and a recording agree about the same thing, you can ask the picture friend to describe the picture, ask the listening friend to say what was said, and then give both descriptions to the reading friend. That works, but something always gets lost on the way. The picture friend might forget to mention that the chart started at 95 instead of 0. The listening friend might not mention that the speaker paused.
A multimodal model is one friend who can look and listen at the same time. It chops a picture into little squares and turns each square into a number. It slices sound into tiny time windows and turns each window into a number. Words already are numbers to it. Then it puts all of those numbers in one long line and reads the whole line at once.
Because everything is in the same line, it can notice that a bar looks like a cliff while the words say "up 40 percent." That is the whole idea. Same line, same attention, one answer.
Mindmap: The Complete Picture
The mindmap places the four architectural ideas at the centre and branches outward to the modalities that feed them, the tokenization step that converts each modality, the task shapes that reward cross-modal reasoning, the known failure modes, and the four cost levers.
UNOP Sound (University 365 Neuroscience Oriented Pedagogy)
Take five minutes to consolidate your memory. Play the isochronous tone track (10Hz alpha frequency) with your eyes closed. Alpha-frequency tones after a learning session support consolidation, helping move what you just learned from short-term to long-term memory.
[Audio player: UNOP Post-Lecture Isochrone (10Hz, 5 minutes)]
Practical Exercise: Test One Model on a Real Cross-Modal Task
This exercise takes about 15 minutes and needs one multimodal model, one chart, and one short recording. Do not skip the comparison step: the difference between the two runs is the lesson.
Step 1: Build the test material
Take a chart from your own work. Deliberately choose one with a non-zero axis origin or an unusual scale. Record 60 seconds of yourself describing that chart, and include one statement that the chart does not support. Keep the recording slightly noisy, as a real meeting recording would be.
Step 2: Run the pipelined version
First, convert each modality to text separately. Have a vision call describe the chart, and a transcription call turn the audio into words. Then pass both text descriptions to a text model with the question: "Does the speaker's description match the chart?" Record the answer.
Step 3: Run the single-model version
Send the chart image and the audio clip to one multimodal model with the same question. Use a moderate resolution so the axis labels stay legible, and state the question precisely.
Step 4: Compare and diagnose
Ask three questions of the two answers.
Did the pipelined version notice the axis origin? Usually the description omits it, because describing a chart rarely means reporting its axis.
Did either version react to how the statement was delivered, such as a hesitation before the number?
Which answer would you act on, and what would you verify before acting?
What to look for
You are looking for the first answer that depends on both modalities at once. Once you see one, you also see why the architecture question is not academic. If neither run finds it, check your resolution and your audio quality before concluding anything about the model.
Applied AI connection
This is the evaluation pattern you will reuse in every multimodal project: build the comparison, run both architectures on your own inputs, and let the measurement choose. A vendor demo cannot do this for you.
Glossary
Term | Definition |
Modality | A type of input or output signal, such as text, image, audio, or video. |
Multimodal model | A model that accepts or produces more than one modality. The useful distinction is whether it reasons across them or only converts them. |
Vision encoder | The component that turns image patches into vectors the transformer can process. |
Patch | A small square region of an image, commonly 16 by 16 or 32 by 32 pixels, treated as one unit by the vision encoder. |
Visual token | One unit of image input after patch encoding is projected into the model's token space. Billed like text tokens. |
Tile | A 512 pixel square used by some vision tokenizers to price high-detail images. |
Late fusion | An architecture that encodes each modality separately and compares the resulting vectors in a shared space. Good for retrieval, not for cross-modal reasoning. |
Adapter | A projection layer that maps a modality encoder's output into the text model's token space. |
Native multimodal | An architecture in which all modalities enter one transformer from the earliest layers, so cross-attention spans modalities throughout the network. |
Cross-modal reasoning | Answering a question that cannot be answered from any single modality alone. |
Interleaved sequence | A single token sequence containing more than one modality, in reading order. |
Spectrogram | A visual representation of sound over time that an audio encoder can process like an image. |
Frame sampling | Reading video as a fixed number of still frames per second rather than as continuous motion. |
Detail level | A request setting that controls how many tokens an image consumes and how much fine detail the model receives. |
Context window | The total token budget for one request, shared across all modalities, instructions, and retrieved text. |
Layout-aware extraction | Reading a document page as image patches, preserving table and column structure, instead of extracting text first. |
Grounding | Tying an answer to a specific, checkable element of the input, such as a frame, a region, or a timestamp. |
First-token latency | The time from sending a request to receiving the first visible or audible output. Distinct from total latency, and usually the number a user perceives. |
Streaming | Sending partial output as it is produced instead of waiting for the complete answer, so perceived latency is lower for the same total work. |
Provenance | The recorded source, capture date, page or timestamp offset and model version for a derived record. Required to cite a result and to invalidate a bad extraction. |
Chunking | Splitting an input into retrieval-sized units. For text, a few hundred tokens with overlap; for audio, a timestamped window; for video, two keys, a transcript and sampled frames. |
Index cost | The number of embeddings a corpus produces. Frame-level indexing can be thirty times the size of transcript-level indexing for the same hour of content. |
Redaction at ingestion | Removing content from the derived representation the retriever searches, rather than filtering at display time. A display-time filter is one bug away from a disclosure. |
Voice as biometric data | In several jurisdictions a voice recording is treated as a biometric identifier, so an audio pipeline carries a different data class from a text pipeline. |
Quiz: TEST YOUR UNDERSTANDING
1. A 1024 by 1024 image sent at high detail to a GPT-4o class model costs approximately how many tokens?
A) 85
B) 170
C) 765
D) 1024
2. What is the defining weakness of a late-fusion architecture?
A) It cannot process images at all
B) It can retrieve across modalities but cannot reason across them
C) It requires video input
D) It cannot produce text output
3. Why can a native audio model respond faster than a transcribe-then-reason pipeline?
A) It uses a smaller language model
B) It caches the transcript on the server
C) Audio stays inside one network, removing the extra round trip between stages
D) It always returns shorter answers
4. Most multimodal systems process video by:
A) Watching it continuously in real time
B) Sampling frames at a fixed rate and treating them as a sequence of images
C) Converting it to a single summary embedding
D) Transcribing the audio track only
5. Your support tool must route tickets using text visible in a screenshot. What is the appropriate choice?
A) A native multimodal model with the whole screen recording
B) An image-capable model at moderate resolution that extracts the visible text
C) A text-only model with no image input
D) A separate vision model chained to a larger vision model
6. A pipelined voice assistant and a native audio model answer the same question. What accounts for most of the latency gap?
A) The native model uses a smaller language model
B) The pipeline pays for separate speech components and four service round trips
C) The native model caches previous answers
D) The pipeline runs on slower hardware
7. In a fifty-item multimodal test set, where should the smallest category be, and why?
A) Clean text in an image, because it is solved
B) Cross-modal questions, because most production questions use one modality at a time
C) Noisy audio, because it is hard to collect
D) Dense layouts, because OCR handles them
8. One hour of audio indexed as 30-second transcript chunks produces about 120 items. The same hour indexed as one frame per second produces about how many?
A) 120
B) 360
C) 1,200
D) 3,600
9. Why must a chunk inherit a permission rather than relying on a display-time filter?
A) Display filters are too slow
B) A chunk inherits the narrowest permission of anything visible or audible in its span, and a filter is one bug away from a disclosure
C) Embeddings cannot be filtered
D) Providers require it
Answers: 1-C, 2-B, 3-C, 4-B, 5-B, 6-B, 7-B, 8-D, 9-B.
Related Resources
U365 INSIDE Publications
Book Essential: Co-Intelligence by Ethan Mollick: The Centaur model and how to divide work between a human and an AI
Lecture: How LLMs Actually Work: Transformers in 20 Minutes: The attention mechanism that multimodal models reuse for every modality
External Resources
GPT-4o System Card (OpenAI): Reports the 232 to 320 millisecond audio response latency and documents the single-network omni architecture: openai.com/index/gpt-4o-system-card
Images and vision (OpenAI API documentation): The patch and tile token arithmetic, including the 85 base plus 170 per 512 pixel tile rule: developers.openai.com/api/docs/guides/images-vision
Attention Is All You Need (Vaswani et al., 2017): The transformer paper whose attention mechanism every multimodal model extends: arxiv.org/abs/1706.03762
Related U365 Lectures (Coming Soon)
The AI Stack 2026: What Every Developer Needs, in the AI Engineering series
Model Quantization: Running LLMs on Your Laptop, in the AI Engineering series
U.Copilot for This Lecture
Use this prompt with your own AI assistant to turn the lecture into a decision for one workflow you own. It follows the UP-Context-Engineering method: context first, then the task, then the constraints, then the output shape.
CONTEXT I am evaluating whether one workflow I own needs a multimodal model or a pipeline of single-modality models. The workflow is: [describe the workflow in two sentences] The inputs it handles are: [text / images / screenshots / documents / audio / video, with volume per day] The decision it must make is: [state the decision] The cost of a wrong answer is: [state the consequence] TASK 1. For each input type, tell me whether the needed answer can be produced from that modality alone. 2. If every answer comes from one modality at a time, specify a pipeline design. 3. If any answer requires two modalities at once, specify a single-model design and name the exact cross-modal question it must answer. 4. For the recommended design, list the resolution or frame-rate setting to test first, and the failure mode that would show the setting is too low. CONSTRAINTS Use only information I have given you plus what is in this conversation. Do not invent benchmark numbers. Where you are uncertain, say which measurement would settle it. OUTPUT A short recommendation, then a table with one row per input type: modality, decision possible from that modality alone (yes or no), and the setting to test first. End with the three measurements I should collect on fifty real items.
Next Steps
Run the practical exercise on one chart and one recording from your own work, and compare the pipelined answer with the single-model answer
Count your image tokens for one real request, using the patch and tile arithmetic, before you commit to a resolution policy
Build a fifty-item eval set from your own inputs, with the sample rate and resolution you can afford in production
Take the next lecture in this institute: "The AI Stack 2026: What Every Developer Needs" for the full modern AI application architecture
Read the GPT-4o system card for the latency and architecture claims in this lecture, and read the image tokenization documentation before you price a vision pipeline
Multimodal capability stops being a mystery the moment you see where the modalities meet. Ask that one question about any system you evaluate: do the modalities meet inside one network, or do they meet in a text description? Your answer predicts the failure modes you will inherit.
IMPORTANT NOTICE
This lecture is published by University 365 as part of its INSIDE Publications Hub. The content is free to read for all visitors. Lectures in this series may be part of a structured academic program leading to a Micro-Credential for your Career (MCC). To enroll in an academic program, visit university-365.com/tuition.
This content is for educational purposes. While we strive for accuracy, AI is a fast-moving field. Verify current technical details against primary sources for professional applications.
Copyright University 365, Inc. All rights reserved. This content is protected under University 365's copyright policies. For permissions or inquiries, contact uda@university-365.com.
Published by the Department of Academics, University 365.
Lecture delivered by the University 365 Institute of Technology (UIT).
Sam Utteker, Dean of Technology, UIT
Signed for the academic year 2026.
Comments