Skip to main content
The following information applies only to Unstructured Pay-As-You-Go and Business SaaS accounts. Multimedia is not available for Let’s Go accounts, or for dedicated instance or in-VPC deployments.
Multimedia is a partitioning strategy for audio and video files. It turns recordings such as support calls, meetings, and training videos into timestamped document elements. These elements flow through the same workflow, embedding, and destinations as elements from documents.

How Multimedia processes audio and video

Multimedia produces different elements for audio and video:
  • Audio becomes transcript fragments. Each TranscriptFragment holds one speaker turn, its start and end times in seconds, and, when available, a speaker label.
  • Video becomes scene descriptions. Unstructured describes video scenes with a default vision model. Each SceneDescription holds a description of the scene, any on-screen text, and the objects and activities shown, with start and end times.
  • Video with a transcribed soundtrack returns both, interleaved in time order. You turn this on through the Pipelines API. See Multimedia settings.
By default, Unstructured processes long files in 5-minute windows. You can set longer windows through the Pipelines API. Each element’s source_window metadata records the window that produced it. Multimedia does not identify people, detect or track objects, translate speech, or produce word-level timestamps. It samples 10 to 20 frames per window, biased toward scene changes, so brief events can fall between frames.

Supported audio and video formats

Unstructured checks each .mp4 and .webm file for a video track. Unstructured processes a file with a video track as video, and a file without one as audio. It rejects files that have no extension.

Build a Multimedia workflow

Unstructured recommends this workflow: a Partitioner node with the Multimedia strategy, then an Embedder node, then your destination. Each transcript fragment and scene description is already a retrieval-sized unit, so do not add a Chunker node. Chunks do not carry time or speaker metadata. To build this workflow in the workflow designer:
1

Add the Partitioner node

Add a Partitioner node directly after the source node, and select Multimedia. The workflow designer shows no other Multimedia settings.
2

Add the Embedder node and connect the destination

Add an Embedder node after the Partitioner node, and connect it to your destination node.
To process one file before you run the whole workflow, use Test in the workflow designer. Multimedia processes only audio and video files, and the Auto strategy does not process them. For a folder that mixes documents and media, create one workflow for documents and one Multimedia workflow for media. To create a Multimedia workflow or run an on-demand job through the Pipelines API, see Multimedia settings.

Output elements

Multimedia returns two element types, sorted by start time:
  • TranscriptFragment: One speaker turn from an audio file or a video’s soundtrack.
  • SceneDescription: A vision model’s description of one video scene.
The following example shows a transcript fragment:
Speaker labels such as SPEAKER_00 are consistent within one file only. They do not identify a person, and they do not match across files. Transcript element IDs stay stable when you reprocess an unchanged file. Scene descriptions come from a vision model, which can divide the same video into scenes differently on each run. Scene element IDs are stable only when the scenes are the same, so reprocessing a video can change its scene IDs. For every metadata field, see Audio and video files.

Multimedia limits

The file-size limit depends on how a file enters the workflow. A file over the size or duration limit fails. Unstructured does not bill for it. For the formats Multimedia accepts, see Supported file types.

Upload rejections

When you upload files through the Pipelines API, Unstructured checks each file’s type before its size, and rejects the whole request before the job runs. For an audio file, the first 415 message says “is an audio file” instead of “is a video file.”

Media minutes usage and billing

Multimedia usage is measured in media minutes, not pages. Media files add no pages to your usage.
  • Unstructured rounds each file’s duration up to the next whole second, then adds it to your minutes.
  • Unstructured bills a video for both audio and video minutes only when transcribe_video_audio is true.
  • The audio rate includes speaker labels.
  • Unstructured does not bill files that fail.
  • Unstructured bills a file with no detectable speech for its full length.
To view media minutes, see Account usage.

Troubleshooting

Common questions

Does Multimedia identify speakers by name? No. Speaker labels distinguish voices within one file. They are not identities. Does Multimedia detect the spoken language? No. You can pass a two-letter language hint through the Pipelines API. The hint labels the output and does not translate it. Can I chunk Multimedia output? Unstructured does not recommend it. Chunks do not carry start_time, end_time, or speaker, and each element is already sized for retrieval. Can Multimedia find every event in a video? No. Multimedia samples frames, so a brief event can fall between them. It describes scenes and does not detect or track objects.