Skip to main content
First time creating a connector? Read this first.

​
Requirements

You will need:

​
Accounts

Unstructured Pipelines and Unstructured API support: Unstructured Ingest supports:

​
Resources

For Weaviate installed locally:
  • The name of the target collection on the local instance.
For Embedded Weaviate:
  • The instance’s connection URL and the name of the target collection on the instance.
For Weaviate Cloud:
  • A Weaviate database instance. The following information assumes that you have a Weaviate Cloud (WCD) account with a Weaviate database cluster in that account. Create a WCD account. Create a database cluster. For other database options, learn more.
  • The URL of the database cluster. Get the URL.
  • The name of the target collection in the database. Create a collection. How you create and specify the collection depends on whether you want document metadata stored as nested JSON in a single column, or have Unstructured flatten the metadata and store each field as a separate column. For more information, see Storing document metadata later in this topic.
    • If you choose to store document metadata as a JSON blob, you can specify a collection, or have the connector attempt to create a collection for you automatically at run time. To specify an existing collection For the connector, specify the name of an existing collection. If you specify an existing collection name, and Unstructured generates embeddings, but the number of dimensions that are generated does not match the existing collection’s embedding settings, the job will fail. You must change your Unstructured embedding settings or your existing collection’s embedding settings to match, and try the job again. To have the connector create a table You can have the connector attempt to create a collection for you automatically at run time. To do so, specify the name of the collection that you want the connector to attempt to create—that is, a collection that does not already exist.
    • If you choose to flatten the document metadata, you must create the collection before you configure your connector. For more information, see Storing document metadata later in this topic.

​
Authentication

  • The API key for the database cluster. For more information, see Authentication in the Weaviate Cloud documentation.

​
Viewing generated embeddings

If Unstructured creates a new collection and generates embeddings, you will not see an embeddings property in tools such as the Weaviate Cloud Collections user interface. To view the generated embeddings, you can run a Weaviate GraphQL query such as the following. In this query, replace <collection-name> with the name of the new collection, and replace <property-name> with the name of each additional available property that you want to return results for, such as text, type, element_id, record_id, and so on. The embeddings will be returned in the vector property. 
{
  Get {
    <collection-name> {
      _additional {
        vector
      }
      <property-name>
      <property-name>
    }
  }
}

​
Inferring missing properties

If auto-schema is enabled in Weaviate (which it is by default), Weaviate can infer missing properties and add them to the collection definition at run time. However, it is a Weaviate best practice to manually define as much of the data schema in advance as possible, since manual definition gives you the most control.

​
Minimal required schema

The minimum viable schema for Unstructured includes only the element_id and record_id properties. The text and type properties should also be included, but they are technically optional.
The record_id, element_id, and id fields are closely related, but each has a distinct purpose. For more information, see How connectors use record IDs, element IDs, and IDs.
The following code example shows how to use the weaviate-client Python package to create a collection in a Weaviate Cloud database cluster with this minimum viable schema, and to specify that Unstructured will generate the embeddings for this collection. To connect to a locally hosted Weaviate instance instead, call weaviate.connect_to_local. To connect to Embedded Weaviate instead, call weaviate.connect_to_embedded. 
import os
import weaviate
from weaviate.classes.init import Auth
import weaviate.classes.config as wvc

client = weaviate.connect_to_weaviate_cloud(
    cluster_url=os.getenv("WEAVIATE_URL"),
    auth_credentials=Auth.api_key(api_key=os.getenv("WEAVIATE_API_KEY")),
)

collection = client.collections.create(
    name="MyCollection",
    properties=[
        wvc.Property(name="element_id", data_type=wvc.DataType.UUID),
        wvc.Property(name="record_id", data_type=wvc.DataType.TEXT),
        wvc.Property(name="text", data_type=wvc.DataType.TEXT),
        wvc.Property(name="type", data_type=wvc.DataType.TEXT),
    ],
    vectorizer_config=None, # Unstructured will generate the embeddings instead of Weaviate.
)

client.close()
Unstructured cannot provide a schema that is guaranteed to work in all circumstances. This is because these schemas will vary based on your source files’ types; how you want Unstructured to partition, chunk, and generate embeddings; any custom post-processing code that you run; and other factors. See also:

​
Storing document metadata

Unstructured offers the following options for storing document metadata in the destination table:
  • Store the metadata as a single nested JSON field: 
    {
    "metadata": {
    "category_depth": 1,
    "data_source": {
        "url": "s3://my-source-bucket/path/chart-and-table.pdf",
    . . .
    }}
}
  • Flatten the metadata by writing each metadata field as its own typed, queryable column: 
    {
    "category_depth": 1,
    "data_source_url": "s3://my-source-bucket/path/chart-and-table.pdf",
    . . .
}
In general, storing the metadata as a JSON blob works for most use cases, unless you want query individual metadata fields directly using standard SQL, or you are using tools that require columnar data. Storing the metadata as a JSON blob works for most use cases, including when:
  • Performing dot.notation queries on the stored JSON is sufficient for your needs.
  • Document metadata schemas vary across file sources. When flattening document metadata, Unstructured drops fields that do not match existing columns in the schema.
  • You want the connector to automatically generate the destination table. This option is not supported when flattening document metadata.
Flattening the metadata and storing it in multiple columns is most useful when:
  • You want to query individual metadata fields directly using standard SQL, without JSON parsing.
  • The business intelligence or analytics tools you are using require columnar data.
To store metadata as a JSON blob, when configuring the connector uncheck Flatten Metadata (in the Unstructured Pipelines), or set flatten_metadata to false (in the Unstructured API). To flatten the metadata, check Flatten Metadata, or set flatten_metadata to true. Storing metadata as a JSON blob is the default.

​
Considerations when flattening metadata

If you choose to have Unstructured flatten document metadata, you must create the collection to use as the destination, and specify it when creating the connector. In order to prevent possible data loss, Unstructured will not automatically create a new collection. Considerations to keep in mind when creating the collection:
  • The collection must contain a column for each metadata field you want to store. Any metadata field that does not have a corresponding column in the table is silently dropped, although the event is written to the logs. For more information, see Logging and monitoring.
  • The collection must contain a record_id property. Unstructured requires this property for re-run deduplication.
  • Do not declare metadata columns as NOT NULL. Missing metadata values are written as NULL.
  • Unstructured passes values through as their JSON-native type: strings, numbers, boolean, and so on. For example, no special formatting is applied to timestamp values.
  • Metadata fields that are lists are not further flattened. Lists remain single columns.
  • Declare any list-of-object fields as of the object array type. For flattened fields that contain lists of objects—such as links, permissions, or regex matches—declare the property type as OBJECT_ARRAY in the collection schema.

​
Metadata flattening example

The following example demonstrates how Unstructured flattens metadata into separate columns. Consider the following metadata: 
{
  "metadata": {
    "category_depth": 1,
    "data_source": {
      "url": "s3://my-source-bucket/path/chart-and-table.pdf",
      "version": "864ae40b0719e976e98ba0a7b9fcba92",
      "record_locator": {
        "protocol": "s3",
        "remote_file_path": "s3://my-source-bucket/path/"
      }
    },
    "languages": ["eng"]
  }
}
When flattening metadata, Unstructured generates a field name comprised of the full path to that field within the metadata structure, from the outermost object to the field itself. For example, protocol, which is included in the record_locator object, which is in turn within data_source, becomes data_source_record_locator_protocol: 
{
  "category_depth": 1,
  "data_source_url": "s3://my-source-bucket/path/chart-and-table.pdf",
  "data_source_version": "864ae40b0719e976e98ba0a7b9fcba92",
  "data_source_record_locator_protocol": "s3",
  "data_source_record_locator_remote_file_path": "s3://my-source-bucket/path/",
  "languages": ["eng"]
}

​
Examples

To create a Weaviate destination connector, see the following examples. For more information on working with destination connectors using the Unstructured API, see Destination endpoints. 
import os

from unstructured_client import UnstructuredClient
from unstructured_client.models.operations import CreateDestinationRequest
from unstructured_client.models.shared import CreateDestinationConnector

with UnstructuredClient(api_key_auth=os.getenv("UNSTRUCTURED_API_KEY")) as client:
    response = client.destinations.create_destination(
        request=CreateDestinationRequest(
            create_destination_connector=CreateDestinationConnector(
                name="<name>",
                type="weaviate-cloud",
                config={
                    "cluster_url": "<host-url>",
                    "collection": "<class-name>",
                    "api_key": "<api-key>",
                    "flatten_metadata": "<true|false>"
                }
            )
        )
    )

    print(response.destination_connector_information)

curl --request 'POST' --location \
"$UNSTRUCTURED_API_URL/destinations" \
--header 'accept: application/json' \
--header "unstructured-api-key: $UNSTRUCTURED_API_KEY" \
--header 'content-type: application/json' \
--data \
'{
    "name": "<name>",
    "type": "weaviate",
    "config": {
        "host_url": "<host-url>",
        "class_name": "<class-name>",
        "api_key": "<api-key>",
        "flatten_metadata": "<true|false>"
    }
}'

​
Configuration settings

Replace the preceding placeholders as follows:
​
name
string
required
A unique name for this connector.
​
host_url
string
required
The URL of the Weaviate database cluster.
​
class_name
string
The name of the target collection within the cluster. If you specify the name of a collection that does not exist, the connector attempts to create it.
​
flatten_metadata
boolean
default:"false"
Set to true to have Unstructured flatten the metadata and store each field as a separate columns, or false to store document metadata as nested JSON in a single column. For more information, see Storing document metadata.
​
api_key
string
required
The API key provided by Weaviate to access the cluster.