Embedding

For the Unstructured open source library version of this page, see Embedding for open source.

After partitioning, chunking, and summarizing, the embedding step creates arrays of numbers known as vectors, representing the text that is extracted by Unstructured. These vectors are stored or embedded next to the text itself. These vector embeddings are generated by an embedding model that is provided by an embedding provider. You typically save these embeddings in a vector store. When a user queries a retrieval-augmented generation (RAG) application, the application can use a vector database to perform a similarity search in that vector store and then return the items whose embeddings are the closest to that user’s query. Here is an example of a document element generated by Unstructured, along with its vector embeddings generated by the embedding model sentence-transformers/all-MiniLM-L6-v2 on Hugging Face:

{
    "type": "Title",
    "element_id": "fdbf5369-4485-453b-9701-1bb42c83b00b",
    "text": "THE CONSTITUTION of the United States",
    "metadata": {
        "filetype": "application/pdf",
        "languages": [
            "eng"
        ],
        "page_number": 1,
        "filename": "constitution.pdf",
        "data_source": {
            "record_locator": {
                "path": "/input/constitution.pdf"
            },
            "date_created": "1723069423.0536132",
            "date_modified": "1723069423.055078",
            "date_processed": "1725666244.571788",
            "permissions_data": [
                { 
                    "mode": 33188
                }
            ]
        }
    },
    "embeddings": [
        -0.06138836592435837,
        0.08634615689516068,
        -0.019471267238259315,
        "<full-results-omitted-for-brevity>",
        0.0895417109131813,
        0.05604064092040062,
        0.01376157347112894
    ]
}

Understanding embedding models: make an informed choice for your RAG.

Generate embeddings

To have Unstructured generate embeddings, do the following:

For Unstructured Pipelines users, add an Embedder node to an Unstructured custom workflow.
For Unstructured API users, add an Embedder node as either as an object in a workflow_nodes array (for curl) or as a WorkflowNode in a WorkflowNodes collection (for Python) whenever you create a workflow, update a workflow, or create a workflow job that processes local files.

When choosing an embedding model, be sure to pay attention to the number of dimensions for the model you choose. This number must match the number of dimensions in the embeddings field of your destination connector’s table, collection, or index.

Chunk sizing and embedding models

If your workflow has an Embedder node, your workflow’s Chunker node settings must stay within the selected embedding model’s token limits. Exceeding these limits will cause workflow failures. Set your Chunker node’s Max Characters to a value at or below Unstructured’s recommended maximum chunk size for your selected embedding model, as listed in the following table’s last column.

The following list applies only to Unstructured Let’s Go and Pay-As-You-Go accounts.For Unstructured Business accounts, see your Unstructured account administrator for your list of available embedding models. To add more embedding models to your list, contact your Unstructured account administrator or Unstructured sales representative, or request support.

Embedding model	Dimensions	Tokens	Chunker Max Characters^*
Amazon Bedrock
Cohere Embed English	1024	512	1792
Cohere Embed Multilingual	1024	512	1792
Titan Embeddings G1 - Text	1536	8192	28672
Titan Multimodal Embeddings G1	1024	256	896
Titan Text Embeddings V2	1024	8192	28672
Azure OpenAI
Text Embedding 3 Large	3072	8192	28672
Text Embedding 3 Small	1536	8192	28672
Text Embedding Ada 002	1536	8192	28672
Together AI
M2-Bert 80M 32K Retrieval	768	8192	28672
Voyage AI
Voyage 3	1024	32000	112000
Voyage 3 Large	1024	32000	112000
Voyage 3 Lite	512	32000	112000
Voyage Code 2	1536	16000	56000
Voyage Code 3	1024	32000	112000
Voyage Finance 2	1024	32000	112000
Voyage Law 2	1024	16000	56000
Voyage Multimodal 3	1024	32000	112000

^* This is an approximate value, determined by multiplying the embedding model’s token limit by 3.5.

Structured data extractor

Enriching

Generate embeddings

Chunk sizing and embedding models

​Generate embeddings

​Chunk sizing and embedding models

Generate embeddings

Chunk sizing and embedding models