Unstructured embedding

After partitioning, chunking, and summarizing, the embedding step creates arrays of numbers known as vectors, representing the text that is extracted by Unstructured. These vectors are stored or embedded next to the text itself. These vector embeddings are generated by an embedding model that is provided by an embedding provider. You typically save these embeddings in a vector store. When a user queries a retrieval-augmented generation (RAG) application, the application can use a vector database to perform a similarity search in that vector store and then return the items whose embeddings are the closest to that user’s query. The following example shows a document element with an illustrative embedding vector:

{
    "type": "Title",
    "element_id": "fdbf5369-4485-453b-9701-1bb42c83b00b",
    "text": "THE CONSTITUTION of the United States",
    "metadata": {
        "filetype": "application/pdf",
        "languages": [
            "eng"
        ],
        "page_number": 1,
        "filename": "constitution.pdf",
        "data_source": {
            "record_locator": {
                "path": "/input/constitution.pdf"
            },
            "date_created": "1723069423.0536132",
            "date_modified": "1723069423.055078",
            "date_processed": "1725666244.571788",
            "permissions_data": [
                { 
                    "mode": 33188
                }
            ]
        }
    },
    "embeddings": [
        -0.06138836592435837,
        0.08634615689516068,
        -0.019471267238259315,
        "<full-results-omitted-for-brevity>",
        0.0895417109131813,
        0.05604064092040062,
        0.01376157347112894
    ]
}

Understanding embedding models: make an informed choice for your RAG.

Generate embeddings

To have Unstructured generate embeddings, do the following:

For Unstructured Pipelines users, add an Embedder node to an Unstructured custom workflow.
For Unstructured API users, add an Embedder node when you create a workflow, update a workflow, or create a workflow job that processes local files. For curl, add an object to the workflow_nodes array. For Python, add a WorkflowNode to a WorkflowNodes collection.

When choosing an embedding model, be sure to pay attention to the number of dimensions for the model you choose. This number must match the number of dimensions in the embeddings field of your destination connector’s table, collection, or index.

Chunk sizing and embedding models

Each embedding model accepts a limited number of tokens per input. Chunks that exceed the limit can cause a workflow failure or produce embeddings that omit text. Set the Chunker node’s Max Characters to the value recommended for your embedding model in the following table. For IBM models, Unstructured truncates each input to the first 2,000 characters.

The following table applies to Unstructured Let’s Go, Pay-As-You-Go, and Business SaaS accounts.For dedicated instance and in-VPC deployments, see AI providers.

For the model IDs used to configure the Embedder node, see Available models.

Embedding model	Dimensions	Tokens	Chunker Max Characters^*
Amazon Bedrock
Cohere Embed English	1024	512	1792
Cohere Embed Multilingual	1024	512	1792
Titan Embeddings G1 - Text	1536	8192	28672
Titan Multimodal Embeddings G1	1024	256	896
Titan Text Embeddings V2	1024	8192	28672
Azure OpenAI
Text Embedding 3 Large	3072	8192	28672
Text Embedding 3 Small	1536	8192	28672
IBM watsonx
Granite Embedding 278M Multilingual	768	512	1792
Multilingual E5 Large	1024	512	1792
Slate 125M English Retriever v2	768	512	1792
Slate 30M English Retriever v2	384	512	1792
Voyage AI
Voyage 3	1024	32000	112000
Voyage 3 Large	1024	32000	112000
Voyage 3 Lite	512	32000	112000
Voyage Code 2	1536	16000	56000
Voyage Code 3	1024	32000	112000
Voyage Finance 2	1024	32000	112000
Voyage Law 2	1024	16000	56000
Voyage Multimodal 3	1024	32000	112000

^* This approximate value equals the embedding model’s token limit multiplied by 3.5.

Extract

Enrich

Generate embeddings

Chunk sizing and embedding models

​Generate embeddings

​Chunk sizing and embedding models

Generate embeddings

Chunk sizing and embedding models