The following information applies only to Unstructured Pay-As-You-Go and Business SaaS accounts. Multimedia is not available for Let’s Go accounts, or for dedicated instance or in-VPC deployments.
How Multimedia processes audio and video
Multimedia produces different elements for audio and video:- Audio becomes transcript fragments. Each
TranscriptFragmentholds one speaker turn, its start and end times in seconds, and, when available, a speaker label. - Video becomes scene descriptions. Unstructured describes video scenes with a default vision model. Each
SceneDescriptionholds a description of the scene, any on-screen text, and the objects and activities shown, with start and end times. - Video with a transcribed soundtrack returns both, interleaved in time order. You turn this on through the Pipelines API. See Multimedia settings.
source_window metadata records the window that produced it.
Multimedia does not identify people, detect or track objects, translate speech, or produce word-level timestamps. It samples 10 to 20 frames per window, biased toward scene changes, so brief events can fall between frames.
Supported audio and video formats
Unstructured checks each
.mp4 and .webm file for a video track. Unstructured processes a file with a video track as video, and a file without one as audio. It rejects files that have no extension.
Build a Multimedia workflow
Unstructured recommends this workflow: a Partitioner node with the Multimedia strategy, then an Embedder node, then your destination. Each transcript fragment and scene description is already a retrieval-sized unit, so do not add a Chunker node. Chunks do not carry time or speaker metadata. To build this workflow in the workflow designer:1
Add the Partitioner node
Add a Partitioner node directly after the source node, and select Multimedia. The workflow designer shows no other Multimedia settings.
2
Add the Embedder node and connect the destination
Add an Embedder node after the Partitioner node, and connect it to your destination node.
Output elements
Multimedia returns two element types, sorted by start time:TranscriptFragment: One speaker turn from an audio file or a video’s soundtrack.SceneDescription: A vision model’s description of one video scene.
SPEAKER_00 are consistent within one file only. They do not identify a person, and they do not match across files.
Transcript element IDs stay stable when you reprocess an unchanged file. Scene descriptions come from a vision model, which can divide the same video into scenes differently on each run. Scene element IDs are stable only when the scenes are the same, so reprocessing a video can change its scene IDs.
For every metadata field, see Audio and video files.
Multimedia limits
The file-size limit depends on how a file enters the workflow.
A file over the size or duration limit fails. Unstructured does not bill for it. For the formats Multimedia accepts, see Supported file types.
Upload rejections
When you upload files through the Pipelines API, Unstructured checks each file’s type before its size, and rejects the whole request before the job runs.
For an audio file, the first
415 message says “is an audio file” instead of “is a video file.”
Media minutes usage and billing
Multimedia usage is measured in media minutes, not pages. Media files add no pages to your usage.- Unstructured rounds each file’s duration up to the next whole second, then adds it to your minutes.
- Unstructured bills a video for both audio and video minutes only when
transcribe_video_audioistrue. - The audio rate includes speaker labels.
- Unstructured does not bill files that fail.
- Unstructured bills a file with no detectable speech for its full length.
Troubleshooting
Common questions
Does Multimedia identify speakers by name? No. Speaker labels distinguish voices within one file. They are not identities. Does Multimedia detect the spoken language? No. You can pass a two-letter language hint through the Pipelines API. The hint labels the output and does not translate it. Can I chunk Multimedia output? Unstructured does not recommend it. Chunks do not carrystart_time, end_time, or speaker, and each element is already sized for retrieval.
Can Multimedia find every event in a video?
No. Multimedia samples frames, so a brief event can fall between them. It describes scenes and does not detect or track objects.
